2021-03-18T16:36:52.883 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-18T16:36:53.006 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi060.front.sepia.ceph.com' reimaging 2021-03-18T16:36:53.007 INFO:teuthology.lock.ops:Updating [ubuntu@smithi060.front.sepia.ceph.com]: reset os type and version on server 2021-03-18T16:36:53.007 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2021-03-18T16:36:53.025 INFO:teuthology.lock.ops:Node 'ubuntu@smithi060.front.sepia.ceph.com' reimaging is complete 2021-03-18T16:36:53.025 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi205.front.sepia.ceph.com' reimaging 2021-03-18T16:36:53.026 INFO:teuthology.lock.ops:Updating [ubuntu@smithi205.front.sepia.ceph.com]: reset os type and version on server 2021-03-18T16:36:53.026 INFO:teuthology.lock.ops:Updating smithi205.front.sepia.ceph.com on lock server 2021-03-18T16:36:53.041 INFO:teuthology.lock.ops:Node 'ubuntu@smithi205.front.sepia.ceph.com' reimaging is complete 2021-03-18T16:36:53.227 INFO:teuthology.provision.fog.smithi205:Scheduling deploy of centos 8.2 2021-03-18T16:36:53.233 INFO:teuthology.provision.fog.smithi060:Scheduling deploy of centos 8.2 2021-03-18T16:36:53.545 INFO:teuthology.orchestra.console:Power off smithi205 2021-03-18T16:36:53.569 INFO:teuthology.orchestra.console:Power off smithi060 2021-03-18T16:37:01.976 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2021-03-18T16:37:02.077 INFO:teuthology.orchestra.console:Power on smithi060 2021-03-18T16:37:06.107 INFO:teuthology.orchestra.console:Power off for smithi205 completed 2021-03-18T16:37:06.209 INFO:teuthology.orchestra.console:Power on smithi205 2021-03-18T16:37:06.333 INFO:teuthology.orchestra.console:Power on for smithi060 completed 2021-03-18T16:37:06.434 INFO:teuthology.provision.fog.smithi060:Waiting for deploy to finish 2021-03-18T16:37:10.483 INFO:teuthology.orchestra.console:Power on for smithi205 completed 2021-03-18T16:37:10.584 INFO:teuthology.provision.fog.smithi205:Waiting for deploy to finish 2021-03-18T16:41:23.380 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-18T16:41:23.690 INFO:teuthology.provision.fog.smithi060:Node is ready 2021-03-18T16:41:23.815 INFO:teuthology.orchestra.run.smithi060.stdout:smithi060.front.sepia.ceph.com 2021-03-18T16:41:23.871 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60 smithi060.front.sepia.ceph.com smithi060 2021-03-18T16:41:24.219 INFO:teuthology.provision.fog.smithi060:Deploy complete! 2021-03-18T16:41:57.503 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-18T16:41:57.797 INFO:teuthology.provision.fog.smithi205:Node is ready 2021-03-18T16:41:57.814 INFO:teuthology.orchestra.run.smithi205.stdout:smithi205.front.sepia.ceph.com 2021-03-18T16:41:57.869 INFO:teuthology.orchestra.run.smithi205.stdout:172.21.15.205 smithi205.front.sepia.ceph.com smithi205 2021-03-18T16:41:58.193 INFO:teuthology.provision.fog.smithi205:Deploy complete! 2021-03-18T16:41:58.333 INFO:teuthology.lock.ops:Checking smithi060.front.sepia.ceph.com 2021-03-18T16:41:58.333 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-18T16:41:58.350 INFO:teuthology.lock.ops:Checking smithi205.front.sepia.ceph.com 2021-03-18T16:41:58.350 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-18T16:41:58.365 INFO:teuthology.lock.ops:Updating [smithi060.front.sepia.ceph.com]: set os type and version on server 2021-03-18T16:41:58.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-18T16:41:58.659 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2021-03-18T16:41:58.715 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Linux" 2021-03-18T16:41:58.715 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="8 (Core)" 2021-03-18T16:41:58.715 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2021-03-18T16:41:58.715 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2021-03-18T16:41:58.715 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="8" 2021-03-18T16:41:58.715 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el8" 2021-03-18T16:41:58.716 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-03-18T16:41:58.716 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2021-03-18T16:41:58.716 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-18T16:41:58.716 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.centos.org/" 2021-03-18T16:41:58.716 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-18T16:41:58.716 INFO:teuthology.orchestra.run.smithi060.stdout: 2021-03-18T16:41:58.716 INFO:teuthology.orchestra.run.smithi060.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-18T16:41:58.716 INFO:teuthology.orchestra.run.smithi060.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-18T16:41:58.716 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-18T16:41:58.716 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-03-18T16:41:58.716 INFO:teuthology.orchestra.run.smithi060.stdout: 2021-03-18T16:41:58.718 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2021-03-18T16:41:58.733 INFO:teuthology.lock.ops:Updating [smithi205.front.sepia.ceph.com]: set os type and version on server 2021-03-18T16:41:58.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-18T16:41:58.998 INFO:teuthology.orchestra.run.smithi205.stdout:x86_64 2021-03-18T16:41:59.052 INFO:teuthology.orchestra.run.smithi205.stdout:NAME="CentOS Linux" 2021-03-18T16:41:59.052 INFO:teuthology.orchestra.run.smithi205.stdout:VERSION="8 (Core)" 2021-03-18T16:41:59.053 INFO:teuthology.orchestra.run.smithi205.stdout:ID="centos" 2021-03-18T16:41:59.053 INFO:teuthology.orchestra.run.smithi205.stdout:ID_LIKE="rhel fedora" 2021-03-18T16:41:59.053 INFO:teuthology.orchestra.run.smithi205.stdout:VERSION_ID="8" 2021-03-18T16:41:59.053 INFO:teuthology.orchestra.run.smithi205.stdout:PLATFORM_ID="platform:el8" 2021-03-18T16:41:59.053 INFO:teuthology.orchestra.run.smithi205.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-03-18T16:41:59.053 INFO:teuthology.orchestra.run.smithi205.stdout:ANSI_COLOR="0;31" 2021-03-18T16:41:59.053 INFO:teuthology.orchestra.run.smithi205.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-18T16:41:59.053 INFO:teuthology.orchestra.run.smithi205.stdout:HOME_URL="https://www.centos.org/" 2021-03-18T16:41:59.053 INFO:teuthology.orchestra.run.smithi205.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-18T16:41:59.054 INFO:teuthology.orchestra.run.smithi205.stdout: 2021-03-18T16:41:59.054 INFO:teuthology.orchestra.run.smithi205.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-18T16:41:59.054 INFO:teuthology.orchestra.run.smithi205.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-18T16:41:59.054 INFO:teuthology.orchestra.run.smithi205.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-18T16:41:59.054 INFO:teuthology.orchestra.run.smithi205.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-03-18T16:41:59.054 INFO:teuthology.orchestra.run.smithi205.stdout: 2021-03-18T16:41:59.055 INFO:teuthology.lock.ops:Updating smithi205.front.sepia.ceph.com on lock server 2021-03-18T16:41:59.125 INFO:teuthology.dispatcher.supervisor:Running job 5977701 2021-03-18T16:41:59.125 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-03-18_16:32:54-rados:cephadm:dashboard-cephadm-iscsi-test-distro-basic-smithi/5977701 --name sage-2021-03-18_16:32:54-rados:cephadm:dashboard-cephadm-iscsi-test-distro-basic-smithi --description rados:cephadm:dashboard/{0-distro/centos_8.2_kubic_stable task/test_e2e} -- /home/teuthworker/archive/sage-2021-03-18_16:32:54-rados:cephadm:dashboard-cephadm-iscsi-test-distro-basic-smithi/5977701/orig.config.yaml 2021-03-18T16:41:59.130 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-03-18_16:32:54-rados:cephadm:dashboard-cephadm-iscsi-test-distro-basic-smithi/5977701 2021-03-18T16:41:59.130 INFO:teuthology.dispatcher.supervisor:Job PID: 11609 2021-03-18T16:41:59.130 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-19T01:40:36.619 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/ Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 498, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 463, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 320, in report_job response.raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/sage-2021-03-18_16:32:54-rados:cephadm:dashboard-cephadm-iscsi-test-distro-basic-smithi/jobs/5977701/ 2021-03-19T04:42:48.571 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T04:42:48.759 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:42:49.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:42:49.813 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T04:42:49.814 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T04:42:49.814 INFO:teuthology.misc:Compressing logs... 2021-03-19T04:42:49.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T04:42:52.866 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T04:42:52.902 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:42:53.853 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:44:53.927 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T04:44:53.964 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:44:54.981 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:44:54.998 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T04:44:54.998 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T04:44:54.998 INFO:teuthology.misc:Compressing logs... 2021-03-19T04:44:54.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T04:44:58.050 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T04:44:58.085 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:44:59.142 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:46:59.287 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T04:46:59.325 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:47:00.448 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:47:00.464 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T04:47:00.464 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T04:47:00.464 INFO:teuthology.misc:Compressing logs... 2021-03-19T04:47:00.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T04:47:03.526 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T04:47:03.561 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:47:04.592 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:49:04.735 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T04:49:04.772 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:49:05.795 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:49:05.812 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T04:49:05.812 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T04:49:05.812 INFO:teuthology.misc:Compressing logs... 2021-03-19T04:49:05.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T04:49:08.866 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T04:49:08.901 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:49:09.920 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:51:10.078 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T04:51:10.114 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:51:11.120 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:51:11.137 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T04:51:11.137 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T04:51:11.137 INFO:teuthology.misc:Compressing logs... 2021-03-19T04:51:11.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T04:51:14.210 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T04:51:14.245 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:51:15.226 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:53:15.363 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T04:53:15.399 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:53:16.494 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:53:16.509 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T04:53:16.510 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T04:53:16.510 INFO:teuthology.misc:Compressing logs... 2021-03-19T04:53:16.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T04:53:19.586 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T04:53:19.622 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:53:20.634 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:55:20.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T04:55:20.815 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:55:21.825 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:55:21.842 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T04:55:21.843 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T04:55:21.843 INFO:teuthology.misc:Compressing logs... 2021-03-19T04:55:21.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T04:55:24.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T04:55:24.932 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:55:25.918 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:57:26.055 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T04:57:26.093 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:57:27.167 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:57:27.183 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T04:57:27.183 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T04:57:27.183 INFO:teuthology.misc:Compressing logs... 2021-03-19T04:57:27.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T04:57:30.242 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T04:57:30.278 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:57:31.296 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:59:31.431 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T04:59:31.464 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:59:32.459 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T04:59:32.476 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T04:59:32.476 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T04:59:32.476 INFO:teuthology.misc:Compressing logs... 2021-03-19T04:59:32.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T04:59:35.554 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T04:59:35.591 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T04:59:36.618 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:01:36.707 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:01:36.761 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:01:37.736 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:01:37.751 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:01:37.751 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:01:37.751 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:01:37.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:01:40.802 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:01:40.836 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:01:41.789 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:03:41.920 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:03:41.956 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:03:42.947 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:03:42.962 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:03:42.963 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:03:42.963 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:03:42.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:03:46.018 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:03:46.053 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:03:47.042 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:05:47.179 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:05:47.214 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:05:48.222 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:05:48.238 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:05:48.238 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:05:48.238 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:05:48.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:05:51.298 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:05:51.333 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:05:52.341 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:07:52.467 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:07:52.501 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:07:53.525 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:07:53.542 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:07:53.542 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:07:53.542 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:07:53.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:07:56.610 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:07:56.645 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:07:57.694 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:09:57.833 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:09:57.870 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:09:58.947 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:09:58.963 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:09:58.963 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:09:58.963 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:09:58.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:10:02.018 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:10:02.054 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:10:03.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:12:03.358 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:12:03.394 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:12:05.233 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:12:05.249 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:12:05.249 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:12:05.249 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:12:05.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:12:08.322 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:12:08.358 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:12:09.366 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:14:09.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:14:09.539 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:14:11.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:14:11.255 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:14:11.255 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:14:11.256 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:14:11.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:14:14.338 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:14:14.376 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:14:16.342 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:16:16.403 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:16:16.438 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:16:18.141 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:16:18.157 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:16:18.157 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:16:18.157 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:16:18.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:16:21.222 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:16:21.258 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:16:23.235 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:18:23.329 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:18:23.366 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:18:25.061 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:18:25.077 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:18:25.077 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:18:25.078 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:18:25.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:18:28.130 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:18:28.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:18:29.370 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:20:29.510 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:20:29.546 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:20:30.712 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:20:30.728 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:20:30.728 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:20:30.728 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:20:30.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:20:33.794 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:20:33.833 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:20:35.959 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:22:36.095 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:22:36.133 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:22:37.951 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:22:37.969 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:22:37.969 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:22:37.969 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:22:37.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:22:38.031 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:22:39.089 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:22:40.188 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:22:41.283 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:22:42.341 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:22:43.441 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:22:44.538 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:22:45.594 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:22:46.691 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:22:47.787 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:22:47.787 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:22:47.826 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:22:49.002 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:24:49.048 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:24:49.084 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:24:50.241 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:24:50.258 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:24:50.258 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:24:50.258 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:24:50.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:24:51.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:24:51.099 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:24:51.103 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:24:52.195 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:24:53.292 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:24:54.422 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:24:55.591 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:24:56.687 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:24:57.783 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:24:58.954 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:25:00.050 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:25:01.146 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:25:01.146 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:25:01.217 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:25:02.408 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:27:02.449 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:27:02.488 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:27:03.549 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:27:03.566 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:27:03.566 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:27:03.566 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:27:03.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:27:03.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:27:03.893 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:27:06.242 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:27:06.282 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:27:07.305 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:29:07.347 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:29:07.391 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:29:08.536 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:29:08.556 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:29:08.556 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:29:08.556 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:29:08.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:29:08.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:29:08.778 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:29:08.986 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:29:09.028 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:29:09.028 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:29:09.067 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:29:09.067 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:29:09.069 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:29:09.134 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T05:29:09.199 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T05:29:09.200 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:29:09.201 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:29:09.230 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T05:29:09.267 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T05:29:09.297 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T05:29:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:29:09.331 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T05:29:09.332 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:29:09.375 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:29:10.675 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:31:10.724 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:31:10.778 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:31:13.149 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:31:13.177 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:31:13.177 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:31:13.185 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:31:13.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:31:13.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:31:13.449 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:31:31.906 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:31:31.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:31:34.557 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:33:34.602 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:33:34.705 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:33:39.754 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:33:39.787 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:33:39.787 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:33:39.788 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:33:39.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:33:40.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:33:40.048 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:33:40.279 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:33:40.313 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:33:40.315 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:33:40.362 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:33:40.362 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:33:40.363 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:33:40.393 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T05:33:40.406 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T05:33:40.509 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:33:45.079 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:35:45.124 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:35:45.333 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:35:51.170 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:35:51.219 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:35:51.219 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:35:51.220 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:35:51.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:35:51.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:35:51.512 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:35:51.767 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:35:51.792 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:35:51.793 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:35:51.835 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:35:51.835 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:35:51.836 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:35:51.864 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T05:35:51.879 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T05:35:51.998 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:35:57.235 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:37:57.287 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:37:57.341 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:38:00.778 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:38:00.850 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:38:00.850 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:38:00.850 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:38:00.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:38:01.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:38:01.154 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:38:01.365 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:38:01.397 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:38:01.398 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:38:01.435 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:38:01.440 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:38:01.441 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:38:01.478 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T05:38:01.485 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T05:38:01.597 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:38:04.542 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:40:04.591 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:40:04.655 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:40:07.084 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:40:07.103 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:40:07.103 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:40:07.103 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:40:07.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:40:07.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:40:07.338 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:40:07.549 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:40:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:40:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:40:07.616 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:40:07.616 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:40:07.617 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:40:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T05:40:07.661 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T05:40:07.705 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:40:08.804 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:42:08.848 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:42:08.885 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:42:09.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:42:09.948 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:42:09.948 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:42:09.948 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:42:09.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:42:10.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:42:10.188 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:42:10.420 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:42:10.421 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:42:10.463 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:42:10.490 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T05:42:10.505 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T05:42:10.545 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:42:11.584 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:44:11.628 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:44:11.666 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:44:12.870 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:44:12.887 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:44:12.888 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:44:12.888 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:44:12.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:44:13.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:44:13.129 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:44:13.365 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:44:13.365 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:44:13.409 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:44:13.436 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T05:44:13.451 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T05:44:13.488 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:44:14.548 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:46:14.590 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:46:14.667 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:46:15.770 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:46:15.787 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:46:15.787 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:46:15.787 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:46:15.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:46:15.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:46:16.021 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:46:16.261 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:46:16.262 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:46:16.303 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:46:16.330 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T05:46:16.345 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T05:46:16.382 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:46:17.518 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:48:17.560 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:48:17.597 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:48:18.662 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:48:18.679 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:48:18.679 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:48:18.679 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:48:18.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:48:18.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:48:18.915 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:48:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:48:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:48:19.199 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:48:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T05:48:19.242 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T05:48:19.279 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:48:20.379 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:50:20.458 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:50:20.497 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:50:21.631 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:50:21.648 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:50:21.648 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:50:21.648 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:50:21.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:50:21.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:50:21.909 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:50:22.135 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:50:22.135 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:50:22.176 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:50:22.209 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T05:50:22.230 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T05:50:22.266 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:50:23.467 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:52:23.516 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:52:23.554 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:52:24.619 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:52:24.636 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:52:24.636 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:52:24.636 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:52:24.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:52:24.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:52:24.881 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:52:43.202 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:52:43.240 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:52:44.290 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:54:44.335 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:54:44.376 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:54:45.508 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:54:45.526 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:54:45.526 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:54:45.526 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:54:45.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:54:45.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:54:45.773 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:54:45.841 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:54:46.942 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:54:48.039 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:54:49.138 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:54:50.239 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:54:51.336 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:54:52.433 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:54:53.531 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:54:54.589 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:54:55.686 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T05:54:55.687 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T05:54:55.731 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:54:56.960 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:56:57.018 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:56:57.062 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:56:58.265 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:56:58.283 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:56:58.283 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:56:58.283 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:56:58.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:56:58.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:56:58.546 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:57:30.607 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:57:30.638 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:57:30.638 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:57:30.691 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:57:30.691 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:57:30.692 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:57:30.717 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T05:57:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T05:57:30.725 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T05:57:30.766 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:57:31.873 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:59:31.918 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T05:59:31.956 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:59:33.050 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T05:59:33.070 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T05:59:33.070 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T05:59:33.070 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:59:33.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:59:33.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T05:59:33.312 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:59:33.522 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T05:59:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:59:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:59:33.589 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T05:59:33.589 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T05:59:33.591 INFO:teuthology.misc:Compressing logs... 2021-03-19T05:59:33.622 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T05:59:33.633 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T05:59:33.675 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T05:59:34.783 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:01:34.825 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:01:34.864 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:01:35.941 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:01:35.960 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:01:35.979 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:01:35.979 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:01:35.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:01:36.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:01:36.219 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T06:01:36.417 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T06:01:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:01:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:01:36.483 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:01:36.483 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:01:36.484 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:01:36.513 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:01:36.527 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:01:36.564 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:01:37.673 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:03:37.720 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:03:37.775 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:03:38.843 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:03:38.860 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:03:38.860 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:03:38.860 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:03:38.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:03:39.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:03:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T06:03:40.066 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T06:03:40.103 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:03:41.105 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:05:41.148 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:05:41.184 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:05:42.251 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:05:42.272 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:05:42.272 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:05:42.272 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:05:42.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:05:42.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:05:42.527 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T06:05:42.781 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T06:05:42.819 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:05:42.819 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:05:42.898 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:05:42.902 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:05:42.903 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:05:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:05:42.937 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:05:42.974 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:05:44.143 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:07:44.189 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:07:44.234 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:07:45.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:07:45.298 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:07:45.298 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:07:45.298 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:07:45.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:07:45.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:07:45.527 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T06:07:45.852 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T06:07:45.887 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:07:45.887 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:07:45.976 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:07:45.976 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:07:45.982 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:07:46.012 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:07:46.013 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:07:46.057 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:07:47.143 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:09:47.188 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:09:47.225 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:09:48.363 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:09:48.380 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:09:48.381 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:09:48.381 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:09:48.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:10:48.383 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T06:10:48.423 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:10:49.534 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:12:49.582 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:12:49.630 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:12:50.819 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:12:50.836 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:12:50.836 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:12:50.837 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:12:50.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:12:50.899 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:12:51.960 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:12:53.060 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:12:54.160 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:12:55.260 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:12:56.363 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:12:57.465 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:12:58.569 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:12:59.667 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:13:00.763 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:13:00.764 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T06:13:00.804 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:13:01.991 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:15:02.049 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:15:02.087 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:15:03.127 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:15:03.186 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:15:03.186 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:15:03.186 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:15:03.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:15:03.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:15:03.564 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:15:03.860 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:15:03.860 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:15:03.963 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:15:03.978 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T06:15:04.061 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:15:04.066 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:15:04.103 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:15:05.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:17:05.246 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:17:05.301 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:17:06.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:17:06.919 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:17:06.919 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:17:06.919 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:17:06.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:17:07.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:17:07.244 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:17:07.531 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:17:07.531 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:17:07.630 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:17:07.644 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T06:17:07.723 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:17:07.727 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:17:07.766 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:17:08.840 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:19:08.880 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:19:08.919 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:19:10.178 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:19:10.196 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:19:10.196 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:19:10.197 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:19:10.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:19:28.583 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T06:19:28.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:19:29.939 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:21:29.984 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:21:30.024 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:21:31.133 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:21:31.150 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:21:31.150 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:21:31.150 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:21:31.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:21:31.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:21:31.529 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:21:31.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:21:31.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:21:31.932 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:21:32.026 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:21:32.032 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:21:32.069 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:21:33.158 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:23:33.204 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:23:33.243 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:23:34.326 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:23:34.343 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:23:34.343 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:23:34.343 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:23:34.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:23:34.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:23:34.664 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:23:34.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:23:34.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:23:35.097 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:23:35.188 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:23:35.193 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:23:35.233 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:23:36.302 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:25:36.345 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:25:36.407 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:25:37.587 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:25:37.607 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:25:37.607 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:25:37.607 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:25:37.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:25:37.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:25:37.953 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:25:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:25:38.243 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:25:38.350 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:25:38.445 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:25:38.451 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:25:38.510 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:25:39.671 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:27:39.714 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:27:39.850 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:27:41.022 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:27:41.042 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:27:41.042 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:27:41.042 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:27:41.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:27:41.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:27:41.417 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:27:41.733 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:27:41.733 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:27:41.834 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:27:41.928 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:27:41.933 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:27:41.973 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:27:43.263 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:29:43.307 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:29:43.363 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:29:44.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:29:44.575 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:29:44.575 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:29:44.575 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:29:44.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:29:44.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:29:44.907 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:29:45.208 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:29:45.208 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:29:45.313 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:29:45.408 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:29:45.413 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:29:45.486 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:29:46.829 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:31:46.882 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:31:47.529 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:31:48.722 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:31:48.739 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:31:48.739 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:31:48.739 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:31:48.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:31:49.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:31:49.095 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:31:49.436 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:31:49.436 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:31:49.533 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:31:49.630 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:31:49.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:31:49.676 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:31:50.755 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:33:50.799 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:33:50.865 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:33:52.003 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:33:52.025 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:33:52.025 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:33:52.026 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:33:52.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:33:52.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:33:52.375 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:33:52.715 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:33:52.715 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:33:52.819 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:33:52.919 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:33:52.924 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:33:52.996 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:33:54.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:35:54.635 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:35:54.697 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:35:55.863 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:35:55.881 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:35:55.881 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:35:55.881 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:35:55.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:35:56.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:35:56.278 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:35:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:35:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:35:56.678 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:35:56.773 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:35:56.779 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:35:56.818 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:35:57.954 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:37:57.996 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:37:58.035 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:37:59.232 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:37:59.253 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:37:59.253 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:37:59.253 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:37:59.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:37:59.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:37:59.604 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:37:59.945 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:37:59.945 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:38:00.059 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:38:00.189 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:38:00.195 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:38:00.249 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:38:01.463 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:40:01.507 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:40:01.551 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:40:02.738 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:40:02.757 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:40:02.757 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:40:02.758 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:40:02.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:40:03.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:40:03.091 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:40:03.444 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:40:03.444 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:40:03.544 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:40:03.636 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:40:03.640 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:40:03.698 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:40:04.792 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:42:04.833 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:42:04.875 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:42:05.981 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:42:06.002 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:42:06.002 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:42:06.002 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:42:06.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:42:06.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:42:06.348 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:42:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:42:06.644 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:42:06.740 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:42:06.836 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:42:06.842 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:42:06.881 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:42:08.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:44:08.083 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:44:08.162 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:44:09.372 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:44:09.400 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:44:09.400 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:44:09.401 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:44:09.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:44:09.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:44:09.763 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:44:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:44:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:44:10.221 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:44:10.321 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T06:44:10.325 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T06:44:10.383 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:44:11.637 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:46:11.691 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:46:11.745 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:46:13.240 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:46:13.258 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:46:13.258 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:46:13.258 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:46:13.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:46:16.322 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T06:46:16.362 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:46:17.543 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:48:17.599 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:48:17.640 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:48:18.787 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:48:18.803 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:48:18.803 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:48:18.803 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:48:18.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:48:18.864 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:48:19.962 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:48:21.060 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:48:22.157 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:48:23.273 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:48:24.374 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:48:25.473 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:48:26.571 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:48:27.667 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:48:28.764 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:48:28.765 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T06:48:28.807 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:48:29.962 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:50:30.023 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:50:30.064 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:50:31.104 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:50:31.121 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:50:31.121 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:50:31.121 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:50:31.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:50:31.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:50:31.467 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:50:31.504 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:50:32.601 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:50:33.699 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:50:34.797 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:50:35.895 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:50:36.993 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:50:38.091 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:50:39.190 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:50:40.291 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:50:41.388 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T06:50:41.388 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T06:50:41.425 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:50:42.507 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:52:42.552 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:52:42.648 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:52:43.793 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:52:43.810 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:52:43.811 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:52:43.811 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:52:43.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:52:44.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:52:44.191 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:52:47.234 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T06:52:47.273 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:52:48.329 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:54:48.369 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:54:48.407 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:54:49.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:54:49.559 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:54:49.559 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:54:49.559 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:54:49.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:55:49.562 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T06:55:49.613 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:55:50.707 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:57:50.753 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:57:50.795 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:57:51.942 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:57:51.959 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:57:51.960 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:57:51.960 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:57:51.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:57:52.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:57:52.303 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:57:52.617 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:57:52.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:57:52.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:57:52.674 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:57:52.675 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:57:52.676 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:57:52.812 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:57:52.827 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T06:57:52.863 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T06:57:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T06:57:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:57:52.924 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T06:57:52.924 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:57:52.964 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:57:54.163 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:59:54.207 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T06:59:54.247 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:59:55.287 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T06:59:55.303 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T06:59:55.304 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T06:59:55.304 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:59:55.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:59:55.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T06:59:55.649 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:59:55.972 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T06:59:55.991 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:59:55.991 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:59:56.030 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T06:59:56.030 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:59:56.033 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:59:56.179 INFO:teuthology.misc:Compressing logs... 2021-03-19T06:59:56.195 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T06:59:56.232 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T06:59:56.256 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T06:59:56.256 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:59:56.292 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T06:59:56.292 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T06:59:56.333 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T06:59:57.387 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:01:57.432 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:01:57.492 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:01:58.666 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:01:58.684 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:01:58.685 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:01:58.685 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:01:58.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:01:59.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:01:59.322 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T07:01:59.382 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:01:59.382 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:01:59.383 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:01:59.525 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:01:59.704 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:02:00.913 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:04:00.958 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:04:01.006 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:04:02.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:04:02.197 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:04:02.197 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:04:02.198 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:04:02.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:04:02.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:04:02.848 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T07:04:02.914 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:04:02.915 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:04:02.916 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:04:03.066 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:04:03.151 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T07:04:03.153 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:04:03.214 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:04:04.588 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:06:04.632 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:06:04.675 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:06:05.776 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:06:05.795 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:06:05.795 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:06:05.795 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:06:05.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:06:06.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:06:06.438 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T07:06:06.494 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:06:06.495 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:06:06.498 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:06:06.643 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:06:06.660 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T07:06:06.688 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:06:06.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:06:07.789 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:08:07.834 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:08:07.872 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:08:08.948 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:08:08.978 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:08:08.979 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:08:08.979 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:08:08.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:08:09.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:08:09.647 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T07:08:09.706 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:08:09.706 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:08:09.708 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:08:09.854 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:08:09.871 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T07:08:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/f60d2b1c-8880-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T07:08:09.907 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:08:09.944 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:08:10.994 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:10:11.036 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:10:11.079 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:10:12.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:10:12.186 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:10:12.186 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:10:12.186 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:10:12.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:10:12.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:10:12.835 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T07:10:12.896 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:10:12.897 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:10:12.899 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:10:13.041 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:10:13.059 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T07:10:13.059 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/f60d2b1c-8880-11eb-909e-001a4aab830c/ceph-osd.6.log.gz already exists; not overwritten 2021-03-19T07:10:13.095 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/b84c35fc-8880-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T07:10:13.095 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T07:10:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/f60d2b1c-8880-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T07:10:13.098 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:10:13.143 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:10:14.267 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:12:14.309 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:12:14.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:12:15.402 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:12:15.419 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:12:15.419 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:12:15.419 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:12:15.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:12:15.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:12:34.242 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T07:12:34.280 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:12:35.355 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:14:35.404 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:14:35.448 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:14:36.484 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:14:36.500 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:14:36.500 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:14:36.501 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:14:36.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:14:36.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:14:39.874 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T07:14:39.911 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:14:40.943 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:16:40.994 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:16:41.031 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:16:42.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:16:42.145 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:16:42.145 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:16:42.145 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:16:42.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:16:42.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:16:44.194 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T07:16:44.232 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:16:45.263 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:18:45.307 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:18:45.351 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:18:46.478 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:18:46.496 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:18:46.496 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:18:46.496 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:18:46.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:18:46.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:18:47.211 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T07:18:47.268 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:18:47.269 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:18:47.271 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:18:47.328 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T07:18:47.388 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T07:18:47.388 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:18:47.391 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:18:47.418 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T07:18:47.418 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/f60d2b1c-8880-11eb-909e-001a4aab830c/ceph-osd.6.log.gz already exists; not overwritten 2021-03-19T07:18:47.418 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/f60d2b1c-8880-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T07:18:47.436 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:18:47.475 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:18:48.592 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:20:48.637 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:20:48.723 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:20:49.776 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:20:49.793 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:20:49.793 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:20:49.793 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:20:49.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:20:50.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:20:50.424 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T07:20:50.489 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:20:50.489 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:20:50.491 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:20:50.547 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T07:20:50.603 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T07:20:50.603 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:20:50.606 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:20:50.643 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T07:20:50.643 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/f60d2b1c-8880-11eb-909e-001a4aab830c/ceph-osd.6.log.gz already exists; not overwritten 2021-03-19T07:20:50.643 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/f60d2b1c-8880-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T07:20:50.649 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:20:50.690 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:20:51.731 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:22:51.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:22:51.817 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:22:53.783 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:22:53.799 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:22:53.800 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:22:53.800 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:22:53.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:22:54.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:22:57.187 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T07:22:57.234 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:22:59.132 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:24:59.176 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:24:59.217 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:25:00.645 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:25:00.664 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:25:00.664 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:25:00.664 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:25:00.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:25:19.170 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T07:25:19.210 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:25:20.483 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:27:20.530 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:27:20.568 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:27:21.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:27:21.719 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:27:21.720 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:27:21.720 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:27:21.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:27:24.770 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T07:27:24.807 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:27:25.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:29:25.945 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:29:25.984 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:29:27.110 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:29:27.127 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:29:27.127 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:29:27.127 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:29:27.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:29:59.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:30:00.163 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:30:00.364 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:30:00.364 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:30:00.384 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:30:00.437 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T07:30:00.685 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T07:30:00.685 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:30:00.722 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:30:00.751 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T07:30:00.816 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T07:30:00.816 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:30:00.920 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:30:02.092 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:32:02.143 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:32:02.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:32:03.280 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:32:03.299 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:32:03.299 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:32:03.299 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:32:03.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:32:03.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:32:03.517 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:32:03.915 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:32:03.915 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:32:03.945 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:32:03.979 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T07:32:04.045 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T07:32:04.045 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:32:04.077 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:32:04.111 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T07:32:04.177 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T07:32:04.178 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:32:04.261 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:32:05.405 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:34:05.449 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:34:05.487 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:34:06.582 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:34:06.599 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:34:06.599 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:34:06.599 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:34:06.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:34:07.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:34:07.296 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:34:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:34:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:34:07.910 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:34:07.938 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T07:34:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T07:34:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:34:08.063 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:34:08.089 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T07:34:08.146 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T07:34:08.147 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:34:08.185 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:34:09.286 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:36:09.334 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:36:09.384 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:36:10.555 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:36:10.573 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:36:10.573 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:36:10.573 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:36:10.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:36:10.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:36:10.810 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:36:11.169 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:36:11.169 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:36:11.202 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:36:11.354 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:36:11.380 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T07:36:11.422 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c419b9b4-8884-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T07:36:11.422 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T07:36:11.424 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:36:11.466 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:36:12.537 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:38:12.581 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:38:12.667 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:38:13.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:38:13.843 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:38:13.843 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:38:13.843 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:38:13.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:38:14.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:38:14.080 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:38:14.410 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:38:14.410 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:38:14.446 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:38:14.479 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T07:38:14.489 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:38:14.531 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:38:15.698 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:40:15.747 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:40:15.790 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:40:16.907 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:40:16.925 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:40:16.925 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:40:16.925 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:40:16.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:40:17.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:40:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:40:17.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:40:17.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:40:17.552 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:40:17.593 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T07:40:17.595 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:40:17.638 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:40:18.854 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:42:18.896 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:42:18.969 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:42:21.186 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:42:21.220 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:42:21.220 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:42:21.220 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:42:21.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:42:21.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:42:21.534 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:42:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:42:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:42:21.876 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:42:21.923 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T07:42:21.924 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:42:21.962 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:42:23.952 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:44:24.005 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:44:24.044 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:44:25.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:44:25.712 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:44:25.712 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:44:25.712 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:44:25.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:44:25.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:44:25.962 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:44:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:44:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:44:26.314 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:44:26.350 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T07:44:26.359 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:44:26.410 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:44:28.294 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:46:28.352 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:46:28.397 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:46:29.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:46:29.955 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:46:29.956 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:46:29.956 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:46:29.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:46:30.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:46:30.204 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:46:30.533 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:46:30.533 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:46:30.566 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:46:30.602 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T07:46:30.611 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:46:30.651 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:46:31.958 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:48:32.017 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:48:32.056 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:48:33.104 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:48:33.121 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:48:33.122 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:48:33.122 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:48:33.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:48:33.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:48:33.361 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:48:33.706 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:48:33.707 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:48:33.736 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:48:33.776 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T07:48:33.782 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:48:33.820 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:48:34.868 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:50:34.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:50:34.952 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:50:36.039 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:50:36.070 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:50:36.071 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:50:36.071 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:50:36.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:50:36.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:50:36.319 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:50:36.652 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:50:36.652 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:50:36.686 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:50:36.721 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T07:50:36.731 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:50:36.769 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:50:37.878 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:52:37.923 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:52:37.974 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:52:39.216 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:52:39.233 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:52:39.233 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:52:39.233 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:52:39.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:52:39.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:52:39.506 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:52:39.873 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:52:39.874 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:52:39.902 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:52:39.947 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T07:52:39.949 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:52:39.988 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:52:41.090 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:54:41.140 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:54:41.181 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:54:42.332 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:54:42.350 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:54:42.351 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:54:42.351 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:54:42.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:54:42.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:54:42.610 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:54:42.928 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:54:42.929 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:54:42.953 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:54:43.002 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T07:54:43.004 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:54:43.068 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:54:44.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:56:44.242 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:56:44.286 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:56:45.400 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:56:45.417 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:56:45.417 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:56:45.417 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:56:45.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:56:45.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:56:45.655 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T07:56:46.041 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T07:56:46.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T07:56:46.075 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:56:46.112 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T07:56:46.123 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T07:56:46.162 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:56:47.233 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:58:47.281 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T07:58:47.320 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:58:48.352 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T07:58:48.370 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T07:58:48.370 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T07:58:48.371 INFO:teuthology.misc:Compressing logs... 2021-03-19T07:58:48.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T07:59:48.374 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T07:59:48.410 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T07:59:49.515 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:01:49.566 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:01:49.605 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:01:50.703 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:01:50.720 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:01:50.720 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:01:50.720 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:01:50.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:02:09.218 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T08:02:09.254 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:02:10.368 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:04:10.415 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:04:10.474 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:04:11.506 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:04:11.523 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:04:11.524 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:04:11.524 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:04:11.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:04:43.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:04:43.842 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T08:05:02.278 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T08:05:02.317 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:05:03.401 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:07:03.448 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:07:03.489 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:07:04.542 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:07:04.562 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:07:04.562 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:07:04.562 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:07:04.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:07:04.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:07:04.800 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T08:07:04.834 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:07:05.929 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:07:07.029 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:07:08.086 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:07:09.183 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:07:10.322 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:07:11.384 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:07:12.442 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:07:13.917 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:07:14.977 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:07:14.978 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T08:07:15.016 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:07:16.117 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:09:16.174 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:09:16.213 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:09:17.339 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:09:17.355 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:09:17.355 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:09:17.355 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:09:17.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:09:17.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:09:17.587 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T08:10:17.565 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T08:10:17.609 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:10:18.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:12:18.766 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:12:18.886 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:12:19.969 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:12:19.985 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:12:19.985 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:12:19.986 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:12:19.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:12:20.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:12:20.528 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T08:12:20.611 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T08:12:20.612 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:12:20.614 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:12:20.668 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T08:12:20.747 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T08:12:20.747 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:12:20.749 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:12:20.804 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T08:12:20.890 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T08:12:20.891 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:12:20.931 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:12:22.023 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:14:22.074 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:14:22.112 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:14:23.240 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:14:23.275 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:14:23.275 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:14:23.276 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:14:23.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:14:23.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:15:23.554 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T08:15:23.599 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:15:24.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:17:24.799 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:17:24.856 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:17:26.372 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:17:26.389 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:17:26.389 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:17:26.389 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:17:26.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:17:26.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:17:26.921 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T08:17:26.979 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T08:17:26.979 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:17:26.981 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:17:27.144 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:17:27.194 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T08:17:27.392 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T08:17:27.392 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:17:27.430 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:17:28.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:19:28.556 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:19:28.595 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:19:29.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:19:29.854 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:19:29.855 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:19:29.855 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:19:29.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:19:30.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:19:30.511 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:19:30.682 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:19:30.735 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.3.38872.log.gz already exists; not overwritten 2021-03-19T08:19:30.744 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.38726.log.gz already exists; not overwritten 2021-03-19T08:19:30.761 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.38808.log.gz already exists; not overwritten 2021-03-19T08:19:30.774 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-19T08:19:30.775 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.1.38638.log.gz already exists; not overwritten 2021-03-19T08:19:30.777 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-19T08:19:30.778 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.0.38771.log.gz already exists; not overwritten 2021-03-19T08:19:30.783 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.38678.log.gz already exists; not overwritten 2021-03-19T08:19:30.785 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:19:30.822 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:19:31.880 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:21:31.923 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:21:31.965 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:21:34.056 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:21:34.078 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:21:34.079 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:21:34.079 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:21:34.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:21:34.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:21:34.705 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:21:34.894 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:21:34.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.3.38872.log.gz already exists; not overwritten 2021-03-19T08:21:34.944 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.38726.log.gz already exists; not overwritten 2021-03-19T08:21:34.947 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.38808.log.gz already exists; not overwritten 2021-03-19T08:21:34.950 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-19T08:21:34.950 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.1.38638.log.gz already exists; not overwritten 2021-03-19T08:21:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-19T08:21:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.0.38771.log.gz already exists; not overwritten 2021-03-19T08:21:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.38678.log.gz already exists; not overwritten 2021-03-19T08:21:34.953 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:21:34.995 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:21:36.987 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:23:37.034 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:23:37.075 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:23:39.100 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:23:39.117 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:23:39.117 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:23:39.118 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:23:39.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:23:39.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:23:39.753 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:23:47.825 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:23:47.873 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.3.38872.log.gz already exists; not overwritten 2021-03-19T08:23:47.879 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.38726.log.gz already exists; not overwritten 2021-03-19T08:23:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.38808.log.gz already exists; not overwritten 2021-03-19T08:23:47.884 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-19T08:23:47.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.1.38638.log.gz already exists; not overwritten 2021-03-19T08:23:47.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-19T08:23:47.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.0.38771.log.gz already exists; not overwritten 2021-03-19T08:23:47.890 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.38678.log.gz already exists; not overwritten 2021-03-19T08:23:47.895 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:23:47.936 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:23:50.592 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:25:50.641 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:25:50.681 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:25:51.783 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:25:51.800 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:25:51.800 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:25:51.800 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:25:51.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:25:52.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:25:52.418 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:25:59.304 INFO:teuthology.orchestra.run.smithi205.stderr:tar: .: file changed as we read it 2021-03-19T08:25:59.892 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:25:59.966 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.3.38872.log.gz already exists; not overwritten 2021-03-19T08:25:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.38726.log.gz already exists; not overwritten 2021-03-19T08:26:00.010 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.38808.log.gz already exists; not overwritten 2021-03-19T08:26:00.024 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-19T08:26:00.025 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.1.38638.log.gz already exists; not overwritten 2021-03-19T08:26:00.033 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-19T08:26:00.035 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.0.38771.log.gz already exists; not overwritten 2021-03-19T08:26:00.049 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.38678.log.gz already exists; not overwritten 2021-03-19T08:26:00.051 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:26:00.090 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:26:01.164 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:28:01.207 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:28:01.264 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:28:03.158 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:28:03.175 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:28:03.175 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:28:03.175 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:28:03.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:28:03.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:28:03.794 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:28:10.480 INFO:teuthology.orchestra.run.smithi205.stderr:tar: ./fsx-image_client.0-parent134: file changed as we read it 2021-03-19T08:28:11.349 INFO:teuthology.orchestra.run.smithi205.stderr:tar: .: file changed as we read it 2021-03-19T08:28:11.384 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:28:11.429 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.49320.log.gz already exists; not overwritten 2021-03-19T08:28:11.452 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.38872.log.gz already exists; not overwritten 2021-03-19T08:28:11.454 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.3.38872.log.gz already exists; not overwritten 2021-03-19T08:28:11.466 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.40712.log.gz already exists; not overwritten 2021-03-19T08:28:11.468 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.50723.log.gz already exists; not overwritten 2021-03-19T08:28:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.38726.log.gz already exists; not overwritten 2021-03-19T08:28:11.503 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.2.40712.log.gz already exists; not overwritten 2021-03-19T08:28:11.503 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.38808.log.gz already exists; not overwritten 2021-03-19T08:28:11.503 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.38726.log.gz already exists; not overwritten 2021-03-19T08:28:11.516 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.50723.log.gz already exists; not overwritten 2021-03-19T08:28:11.533 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-19T08:28:12.030 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.1.38638.log.gz already exists; not overwritten 2021-03-19T08:28:12.041 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-19T08:28:12.068 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.0.38771.log.gz already exists; not overwritten 2021-03-19T08:28:12.070 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.49320.log.gz already exists; not overwritten 2021-03-19T08:28:12.071 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.38678.log.gz already exists; not overwritten 2021-03-19T08:28:12.092 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:28:12.169 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:28:13.772 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:30:13.819 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:30:14.019 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:30:16.008 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:30:16.028 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:30:16.028 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:30:16.029 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:30:16.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:30:16.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:30:16.648 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:30:16.849 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:30:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.49320.log.gz already exists; not overwritten 2021-03-19T08:30:16.880 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.38872.log.gz already exists; not overwritten 2021-03-19T08:30:16.880 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.3.38872.log.gz already exists; not overwritten 2021-03-19T08:30:16.883 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.40712.log.gz already exists; not overwritten 2021-03-19T08:30:16.884 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.50723.log.gz already exists; not overwritten 2021-03-19T08:30:16.884 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.38726.log.gz already exists; not overwritten 2021-03-19T08:30:16.885 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.2.40712.log.gz already exists; not overwritten 2021-03-19T08:30:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.38808.log.gz already exists; not overwritten 2021-03-19T08:30:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.38726.log.gz already exists; not overwritten 2021-03-19T08:30:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.50723.log.gz already exists; not overwritten 2021-03-19T08:30:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-19T08:30:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.1.38638.log.gz already exists; not overwritten 2021-03-19T08:30:16.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-19T08:30:16.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.0.38771.log.gz already exists; not overwritten 2021-03-19T08:30:16.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.49320.log.gz already exists; not overwritten 2021-03-19T08:30:16.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.38678.log.gz already exists; not overwritten 2021-03-19T08:30:16.892 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:30:16.941 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:30:18.365 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:32:18.412 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:32:18.616 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:32:20.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:32:20.511 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:32:20.511 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:32:20.511 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:32:20.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:32:20.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:32:21.149 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:32:21.360 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:32:21.393 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.49320.log.gz already exists; not overwritten 2021-03-19T08:32:21.394 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.38872.log.gz already exists; not overwritten 2021-03-19T08:32:21.394 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.3.38872.log.gz already exists; not overwritten 2021-03-19T08:32:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.40712.log.gz already exists; not overwritten 2021-03-19T08:32:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.50723.log.gz already exists; not overwritten 2021-03-19T08:32:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.38726.log.gz already exists; not overwritten 2021-03-19T08:32:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.2.40712.log.gz already exists; not overwritten 2021-03-19T08:32:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.38808.log.gz already exists; not overwritten 2021-03-19T08:32:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.38726.log.gz already exists; not overwritten 2021-03-19T08:32:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.50723.log.gz already exists; not overwritten 2021-03-19T08:32:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-19T08:32:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.1.38638.log.gz already exists; not overwritten 2021-03-19T08:32:21.412 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-19T08:32:21.412 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.0.38771.log.gz already exists; not overwritten 2021-03-19T08:32:21.413 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.49320.log.gz already exists; not overwritten 2021-03-19T08:32:21.413 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.38678.log.gz already exists; not overwritten 2021-03-19T08:32:21.417 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:32:21.461 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:32:23.268 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:34:23.326 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:34:23.364 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:34:25.406 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:34:25.440 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:34:25.441 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:34:25.441 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:34:25.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:34:25.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:34:26.107 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:34:26.170 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T08:34:26.264 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T08:34:26.264 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:34:26.267 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:34:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.49320.log.gz already exists; not overwritten 2021-03-19T08:34:26.316 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.38872.log.gz already exists; not overwritten 2021-03-19T08:34:26.317 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.3.38872.log.gz already exists; not overwritten 2021-03-19T08:34:26.336 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.40712.log.gz already exists; not overwritten 2021-03-19T08:34:26.337 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.50723.log.gz already exists; not overwritten 2021-03-19T08:34:26.337 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.38726.log.gz already exists; not overwritten 2021-03-19T08:34:26.346 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.2.40712.log.gz already exists; not overwritten 2021-03-19T08:34:26.346 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.38808.log.gz already exists; not overwritten 2021-03-19T08:34:26.346 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.38726.log.gz already exists; not overwritten 2021-03-19T08:34:26.348 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.1.50723.log.gz already exists; not overwritten 2021-03-19T08:34:26.350 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-19T08:34:26.353 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.1.38638.log.gz already exists; not overwritten 2021-03-19T08:34:26.360 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-19T08:34:26.361 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster2-client.mirror.0.38771.log.gz already exists; not overwritten 2021-03-19T08:34:26.361 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.49320.log.gz already exists; not overwritten 2021-03-19T08:34:26.361 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cluster1-client.mirror.2.38678.log.gz already exists; not overwritten 2021-03-19T08:34:26.364 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:34:26.410 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:34:27.569 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:36:27.617 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:36:27.669 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:36:28.789 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:36:28.806 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:36:28.806 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:36:28.806 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:36:28.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:36:29.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:37:00.580 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:37:01.678 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:37:02.777 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:37:03.874 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:37:04.971 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:37:06.032 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:37:07.129 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:37:08.227 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:37:09.325 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:37:10.464 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T08:37:10.464 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T08:37:10.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:37:11.628 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:39:11.675 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:39:11.717 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:39:13.591 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:39:13.609 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:39:13.609 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:39:13.609 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:39:13.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:39:13.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:40:13.844 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T08:40:13.893 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:40:15.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:42:15.892 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:42:15.963 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:42:17.886 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:42:17.904 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:42:17.904 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:42:17.904 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:42:17.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:42:20.966 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T08:42:21.006 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:42:22.748 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:44:22.793 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:44:22.832 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:44:24.334 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:44:24.363 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:44:24.364 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:44:24.364 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:44:24.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:44:24.834 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T08:44:24.917 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:44:26.057 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:46:26.102 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:46:26.268 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:46:27.484 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:46:27.573 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:46:27.573 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:46:27.573 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:46:27.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:46:27.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:46:27.821 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T08:46:28.113 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T08:46:28.196 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T08:46:28.197 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:46:28.214 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T08:46:28.214 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:46:28.216 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:46:28.243 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T08:46:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T08:46:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:46:28.351 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:46:28.381 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T08:46:28.404 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T08:46:28.449 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T08:46:28.449 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:46:28.465 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T08:46:28.465 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:46:28.504 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:46:29.584 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:48:29.633 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:48:29.672 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:48:30.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:48:30.720 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:48:30.720 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:48:30.720 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:48:30.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:48:30.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:48:30.959 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T08:48:31.254 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T08:48:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T08:48:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:48:31.312 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T08:48:31.312 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:48:31.315 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:48:31.479 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:48:31.503 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T08:48:31.530 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T08:48:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T08:48:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:48:31.590 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T08:48:31.590 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:48:31.629 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:48:32.629 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:50:32.673 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:50:32.711 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:50:33.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:50:33.829 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:50:33.829 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:50:33.829 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:50:33.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:50:34.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:50:34.065 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T08:50:34.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T08:50:34.391 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:50:34.426 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:50:34.457 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T08:50:34.468 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:50:34.507 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:50:35.576 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:52:35.623 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:52:35.660 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:52:36.645 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:52:36.662 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:52:36.662 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:52:36.662 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:52:36.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:52:36.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:52:36.901 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T08:52:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T08:52:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T08:52:37.257 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:52:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T08:52:37.301 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:52:37.338 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:52:38.393 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:54:38.442 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:54:38.484 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:54:39.585 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:54:39.603 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:54:39.603 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:54:39.603 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:54:39.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:54:39.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:54:40.265 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:54:40.299 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T08:54:40.308 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:54:40.345 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:54:41.447 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:56:41.493 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:56:41.544 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:56:42.735 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:56:42.754 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:56:42.755 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:56:42.755 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:56:42.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:56:42.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:56:43.388 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:56:43.423 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T08:56:43.431 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:56:43.472 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:56:44.588 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:58:44.632 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T08:58:44.674 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:58:45.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T08:58:45.793 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T08:58:45.794 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T08:58:45.794 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:58:45.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:58:46.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T08:58:46.425 INFO:teuthology.misc:Compressing logs... 2021-03-19T08:58:46.465 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T08:58:46.469 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T08:58:46.511 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T08:58:47.633 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:00:47.679 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:00:47.718 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:00:48.988 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:00:49.006 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:00:49.007 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:00:49.007 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:00:49.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:00:49.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:00:49.623 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:00:49.665 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:00:49.667 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:00:49.707 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:00:50.829 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:02:50.881 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:02:50.920 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:02:51.998 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:02:52.016 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:02:52.016 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:02:52.016 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:02:52.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:02:52.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:02:52.617 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:02:52.659 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:02:52.661 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:02:52.701 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:02:53.750 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:04:53.796 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:04:53.834 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:04:54.871 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:04:54.889 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:04:54.890 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:04:54.890 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:04:54.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:04:55.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:04:55.491 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:04:55.530 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:04:55.533 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:04:55.570 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:04:56.592 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:06:56.645 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:06:56.682 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:06:57.725 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:06:57.743 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:06:57.743 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:06:57.743 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:06:57.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:06:57.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:06:58.364 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:06:58.405 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:06:58.409 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:06:58.446 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:06:59.529 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:08:59.570 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:08:59.610 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:09:00.653 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:09:00.671 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:09:00.671 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:09:00.671 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:09:00.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:09:00.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:09:01.292 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:09:01.335 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:09:01.337 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:09:01.376 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:09:02.455 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:11:02.501 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:11:02.539 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:11:03.582 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:11:03.599 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:11:03.599 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:11:03.599 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:11:03.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:11:36.491 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T09:11:36.528 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:11:37.557 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:13:37.607 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:13:37.645 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:13:38.680 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:13:38.697 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:13:38.697 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:13:38.698 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:13:38.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:14:11.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:14:11.709 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T09:14:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T09:14:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:14:11.969 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:14:12.209 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T09:14:12.275 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T09:14:12.275 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:14:12.308 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:14:12.341 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T09:14:12.406 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T09:14:12.406 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:14:12.486 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:14:13.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:16:13.543 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:16:13.600 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:16:14.670 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:16:14.687 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:16:14.687 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:16:14.688 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:16:14.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:16:14.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:16:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T09:16:15.266 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T09:16:15.266 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:16:15.301 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:16:15.467 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:16:15.497 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T09:16:15.560 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T09:16:15.561 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:16:15.975 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:16:17.126 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:18:17.258 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:18:17.299 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:18:18.368 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:18:18.387 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:18:18.387 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:18:18.388 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:18:18.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:18:18.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:18:18.627 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T09:18:18.957 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T09:18:18.957 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:18:18.993 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:18:19.027 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:18:19.037 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:18:19.073 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:18:20.150 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:20:20.202 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:20:20.244 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:20:21.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:20:21.368 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:20:21.368 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:20:21.368 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:20:21.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:20:21.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:20:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T09:20:21.937 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T09:20:21.937 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:20:21.973 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:20:22.005 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:20:22.017 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:20:22.095 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:20:23.155 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:22:23.198 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:22:23.237 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:22:24.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:22:24.383 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:22:24.383 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:22:24.383 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:22:24.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:22:24.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:22:25.013 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:22:25.053 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:22:25.057 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:22:25.098 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:22:26.172 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:24:26.218 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:24:26.261 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:24:27.298 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:24:27.314 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:24:27.314 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:24:27.314 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:24:27.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:24:27.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:24:27.927 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:24:27.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:24:27.973 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:24:28.012 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:24:29.139 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:26:29.183 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:26:29.222 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:26:30.318 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:26:30.340 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:26:30.340 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:26:30.340 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:26:30.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:26:30.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:26:30.973 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:26:31.016 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:26:31.018 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:26:31.057 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:26:32.106 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:28:32.196 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:28:32.233 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:28:33.359 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:28:33.375 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:28:33.375 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:28:33.376 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:28:33.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:28:33.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:28:34.005 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:28:34.041 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:28:34.049 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:28:34.086 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:28:35.294 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:30:35.355 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:30:35.390 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:30:36.445 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:30:36.463 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:30:36.463 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:30:36.463 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:30:36.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:30:36.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:30:37.113 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:30:37.153 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:30:37.157 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:30:37.195 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:30:38.229 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:32:38.270 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:32:38.368 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:32:39.437 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:32:39.454 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:32:39.454 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:32:39.455 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:32:39.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:33:39.458 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T09:33:39.494 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:33:41.346 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:35:41.388 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:35:41.499 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:35:43.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:35:43.353 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:35:43.353 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:35:43.353 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:35:43.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:36:43.357 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T09:36:43.397 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:36:45.279 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:38:45.324 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:38:45.400 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:38:47.463 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:38:47.481 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:38:47.481 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:38:47.481 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:38:47.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:38:47.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:38:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T09:38:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T09:38:48.061 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:38:48.082 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:38:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T09:38:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T09:38:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:38:48.245 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:38:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T09:38:48.335 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T09:38:48.336 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:38:48.627 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:38:50.465 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:40:50.513 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:40:50.550 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:40:51.614 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:40:51.631 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:40:51.631 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:40:51.631 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:40:51.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:41:26.087 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T09:41:26.125 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:41:27.219 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:43:27.266 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:43:27.304 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:43:28.397 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:43:28.414 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:43:28.414 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:43:28.414 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:43:28.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:43:28.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:43:28.663 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T09:43:29.007 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T09:43:29.008 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:43:29.030 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:43:29.206 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:43:29.231 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T09:43:29.297 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T09:43:29.297 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:43:29.591 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:43:30.751 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:45:30.796 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:45:30.866 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:45:31.914 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:45:31.931 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:45:31.931 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:45:31.931 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:45:31.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:45:32.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:45:32.169 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T09:45:32.546 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T09:45:32.546 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:45:32.580 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:45:32.613 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:45:32.625 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:45:32.667 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:45:33.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:47:33.765 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:47:33.803 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:47:34.916 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:47:34.933 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:47:34.933 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:47:34.933 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:47:34.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:47:35.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:47:35.550 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:47:35.585 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:47:35.593 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:47:35.631 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:47:36.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:49:36.750 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:49:36.792 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:49:37.888 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:49:37.907 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:49:37.908 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:49:37.908 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:49:37.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:49:38.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:49:38.525 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:49:38.569 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:49:38.571 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:49:38.612 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:49:39.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:51:39.796 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:51:39.833 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:51:41.760 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:51:41.777 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:51:41.777 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:51:41.777 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:51:41.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:51:41.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:51:42.396 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:51:42.435 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T09:51:42.441 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T09:51:42.480 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:51:44.254 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:53:44.300 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:53:44.340 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:53:45.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:53:45.445 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:53:45.446 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:53:45.446 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:53:45.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:53:48.514 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T09:53:48.555 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:53:49.663 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:55:49.706 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:55:49.763 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:55:50.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:55:50.809 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:55:50.809 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:55:50.809 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:55:50.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:55:50.872 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T09:55:51.973 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T09:55:53.072 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T09:55:54.170 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T09:55:55.268 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T09:55:56.364 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T09:55:57.462 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T09:55:58.561 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T09:55:59.658 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T09:56:00.756 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T09:56:00.756 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T09:56:00.803 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:56:03.039 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:58:03.082 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T09:58:03.143 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:58:05.022 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T09:58:05.040 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T09:58:05.040 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T09:58:05.040 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:58:05.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:58:05.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:58:05.656 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T09:58:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T09:58:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:58:05.959 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:58:05.973 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T09:58:06.026 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T09:58:06.026 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:58:06.091 INFO:teuthology.misc:Compressing logs... 2021-03-19T09:58:06.104 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T09:58:06.159 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T09:58:06.159 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T09:58:06.448 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T09:58:08.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:00:08.279 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:00:08.613 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:00:10.637 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:00:10.656 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:00:10.657 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:00:10.657 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:00:10.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:00:10.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:00:10.970 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T10:00:11.309 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T10:00:11.309 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T10:00:11.352 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:00:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T10:00:11.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T10:00:11.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T10:00:11.494 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:00:11.503 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T10:00:11.559 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T10:00:11.559 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T10:00:11.837 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:00:12.958 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:02:13.003 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:02:13.181 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:02:14.287 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:02:14.304 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:02:14.304 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:02:14.304 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:02:14.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:02:32.770 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:02:32.810 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:02:33.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:04:33.895 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:04:34.007 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:04:35.990 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:04:36.007 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:04:36.008 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:04:36.008 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:04:36.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:04:36.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:04:36.330 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T10:04:36.685 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T10:04:36.685 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T10:04:36.734 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:04:36.878 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:04:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T10:04:36.950 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T10:04:36.950 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T10:04:37.184 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:04:39.109 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:06:39.189 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:06:39.253 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:06:41.160 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:06:41.176 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:06:41.176 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:06:41.177 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:06:41.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:06:41.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:06:41.508 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T10:06:41.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T10:06:41.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T10:06:41.900 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:06:42.061 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:06:42.076 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T10:06:42.134 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T10:06:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T10:06:42.615 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:06:43.763 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:08:43.809 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:08:43.878 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:08:45.838 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:08:45.915 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:08:45.915 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:08:45.916 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:08:45.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:08:46.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:08:46.634 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:08:46.786 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:08:47.232 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:08:48.295 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:10:48.347 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:10:48.412 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:10:50.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:10:50.447 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:10:50.447 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:10:50.447 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:10:50.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:10:50.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:10:51.192 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:10:51.350 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:10:51.925 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:10:53.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:12:53.861 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:12:53.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:12:55.908 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:12:55.934 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:12:55.935 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:12:55.935 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:12:55.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:12:56.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:12:56.641 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:12:56.794 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:12:56.808 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-19T10:12:56.809 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-19T10:12:56.810 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T10:12:56.853 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:12:58.905 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:14:58.954 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:14:59.026 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:15:00.941 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:15:00.958 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:15:00.958 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:15:00.959 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:15:00.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:15:01.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:15:01.673 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:15:01.828 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:15:01.846 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-19T10:15:01.846 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-19T10:15:01.857 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T10:15:01.895 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:15:04.121 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:17:04.176 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:17:04.236 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:17:06.119 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:17:06.135 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:17:06.135 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:17:06.135 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:17:06.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:17:06.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:17:06.903 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:17:07.045 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:17:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-19T10:17:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-19T10:17:07.077 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T10:17:07.116 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:17:08.970 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:19:09.015 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:19:09.052 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:19:10.086 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:19:10.102 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:19:10.103 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:19:10.103 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:19:10.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:19:28.578 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:19:28.617 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:19:29.619 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:21:29.666 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:21:29.732 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:21:31.002 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:21:31.019 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:21:31.019 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:21:31.019 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:21:31.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:21:34.082 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:21:34.122 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:21:35.249 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:23:35.294 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:23:35.357 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:23:37.315 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:23:37.333 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:23:37.334 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:23:37.334 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:23:37.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:23:40.387 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:23:40.424 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:23:42.310 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:25:42.355 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:25:42.426 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:25:44.330 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:25:44.348 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:25:44.348 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:25:44.348 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:25:44.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:25:47.426 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:25:47.467 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:25:49.334 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:27:49.383 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:27:49.421 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:27:50.596 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:27:50.613 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:27:50.614 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:27:50.614 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:27:50.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:27:53.666 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:27:53.705 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:27:54.742 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:29:54.786 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:29:54.877 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:29:56.742 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:29:56.759 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:29:56.759 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:29:56.760 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:29:56.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:29:59.810 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:29:59.873 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:30:01.956 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:32:02.006 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:32:02.046 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:32:03.083 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:32:03.101 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:32:03.101 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:32:03.101 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:32:03.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:32:06.178 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:32:06.215 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:32:07.259 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:34:07.304 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:34:07.341 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:34:08.598 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:34:08.619 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:34:08.619 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:34:08.619 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:34:08.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:34:11.682 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:34:11.716 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:34:12.733 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:36:12.778 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:36:12.858 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:36:13.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:36:13.946 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:36:13.946 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:36:13.946 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:36:13.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:36:17.026 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:36:17.063 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:36:18.181 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:38:18.225 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:38:18.264 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:38:19.317 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:38:19.334 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:38:19.334 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:38:19.335 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:38:19.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:38:22.402 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:38:22.439 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:38:23.483 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:40:23.529 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:40:23.567 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:40:24.624 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:40:24.641 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:40:24.641 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:40:24.641 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:40:24.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:40:27.714 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:40:27.751 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:40:28.813 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:42:28.863 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:42:28.900 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:42:29.924 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:42:29.940 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:42:29.941 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:42:29.941 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:42:29.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:42:32.994 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:42:33.039 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:42:34.116 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:44:34.166 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:44:34.207 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:44:35.316 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:44:35.333 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:44:35.334 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:44:35.334 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:44:35.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:44:38.402 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:44:38.440 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:44:39.458 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:46:39.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:46:39.666 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:46:40.771 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:46:40.789 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:46:40.789 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:46:40.789 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:46:40.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:46:43.846 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:46:43.885 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:46:44.963 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:48:45.007 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:48:45.047 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:48:47.022 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:48:47.038 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:48:47.039 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:48:47.039 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:48:47.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:48:50.120 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:48:50.161 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:48:52.048 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:50:52.094 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:50:52.129 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:50:53.177 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:50:53.193 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:50:53.193 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:50:53.194 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:50:53.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:50:56.258 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:50:56.296 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:50:57.347 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:52:57.391 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:52:57.433 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:52:59.316 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:52:59.338 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:52:59.338 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:52:59.339 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:52:59.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:53:02.406 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:53:02.449 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:53:04.412 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:55:04.456 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:55:04.536 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:55:06.424 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:55:06.441 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:55:06.441 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:55:06.441 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:55:06.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:55:09.510 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:55:09.548 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:55:11.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:57:11.312 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:57:11.381 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:57:12.630 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:57:12.646 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:57:12.646 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:57:12.646 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:57:12.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:57:15.714 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:57:15.753 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:57:16.846 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:59:16.894 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T10:59:16.931 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:59:17.969 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T10:59:17.986 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T10:59:17.986 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T10:59:17.987 INFO:teuthology.misc:Compressing logs... 2021-03-19T10:59:17.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T10:59:21.059 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T10:59:21.093 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T10:59:22.110 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:01:22.152 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:01:22.223 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:01:24.422 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:01:24.440 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:01:24.441 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:01:24.441 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:01:24.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:01:27.526 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:01:27.569 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:01:28.692 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:03:28.737 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:03:28.775 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:03:30.801 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:03:30.818 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:03:30.818 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:03:30.819 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:03:30.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:03:33.890 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:03:33.930 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:03:35.920 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:05:35.964 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:05:36.002 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:05:37.391 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:05:37.407 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:05:37.408 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:05:37.408 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:05:37.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:05:40.487 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:05:40.531 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:05:41.628 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:07:41.672 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:07:41.712 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:07:43.596 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:07:43.613 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:07:43.613 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:07:43.613 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:07:43.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:07:46.695 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:07:46.736 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:07:48.696 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:09:48.744 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:09:48.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:09:49.847 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:09:49.863 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:09:49.864 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:09:49.864 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:09:49.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:09:52.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:09:52.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:09:54.027 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:11:54.070 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:11:54.164 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:11:55.988 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:11:56.005 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:11:56.006 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:11:56.006 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:11:56.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:11:59.074 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:11:59.112 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:12:00.997 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:14:01.041 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:14:01.080 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:14:03.257 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:14:03.277 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:14:03.277 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:14:03.277 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:14:03.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:14:06.338 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:14:06.376 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:14:08.212 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:16:08.272 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:16:08.356 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:16:09.415 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:16:09.432 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:16:09.432 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:16:09.433 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:16:09.433 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:16:12.514 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:16:12.558 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:16:13.634 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:18:13.682 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:18:13.734 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:18:15.677 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:18:15.694 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:18:15.695 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:18:15.695 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:18:15.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:18:18.754 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:18:18.792 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:18:20.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:20:20.407 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:20:20.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:20:21.653 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:20:21.672 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:20:21.672 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:20:21.672 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:20:21.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:20:24.738 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:20:24.775 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:20:25.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:22:25.859 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:22:25.904 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:22:27.041 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:22:27.058 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:22:27.058 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:22:27.059 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:22:27.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:22:30.114 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:22:30.153 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:22:31.258 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:24:31.303 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:24:31.346 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:24:32.410 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:24:32.426 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:24:32.426 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:24:32.426 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:24:32.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:24:35.490 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:24:35.532 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:24:37.242 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:26:37.290 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:26:37.344 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:26:38.434 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:26:38.454 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:26:38.455 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:26:38.455 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:26:38.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:26:41.506 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:26:41.543 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:26:42.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:28:42.696 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:28:42.761 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:28:44.035 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:28:44.052 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:28:44.052 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:28:44.052 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:28:44.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:28:47.106 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:28:47.144 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:28:48.739 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:30:48.785 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:30:48.825 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:30:49.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:30:50.014 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:30:50.014 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:30:50.014 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:30:50.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:30:53.094 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:30:53.131 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:30:54.207 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:32:54.253 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:32:54.293 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:32:55.310 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:32:55.327 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:32:55.327 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:32:55.328 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:32:55.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:32:58.406 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:32:58.443 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:32:59.495 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:34:59.544 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:35:00.092 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:35:01.163 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:35:01.180 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:35:01.180 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:35:01.180 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:35:01.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:35:04.258 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:35:04.308 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:35:05.459 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:37:05.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:37:05.550 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:37:07.562 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:37:07.587 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:37:07.587 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:37:07.588 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:37:07.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:37:10.658 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:37:10.703 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:37:12.520 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:39:12.567 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:39:12.606 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:39:14.441 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:39:14.460 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:39:14.461 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:39:14.461 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:39:14.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:39:17.538 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:39:17.576 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:39:18.694 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:41:18.739 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:41:18.778 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:41:19.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:41:19.869 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:41:19.869 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:41:19.869 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:41:19.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:41:22.946 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:41:23.000 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:41:24.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:43:24.247 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:43:24.287 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:43:25.350 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:43:25.368 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:43:25.368 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:43:25.368 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:43:25.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:43:28.455 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:43:28.491 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:43:29.536 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:45:29.583 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:45:29.623 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:45:30.633 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:45:30.649 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:45:30.649 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:45:30.649 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:45:30.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:45:33.730 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:45:33.768 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:45:34.774 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:47:34.819 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:47:34.861 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:47:35.924 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:47:35.942 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:47:35.942 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:47:35.942 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:47:35.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:47:39.014 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:47:39.052 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:47:40.118 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:49:40.164 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:49:40.205 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:49:41.285 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:49:41.317 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:49:41.317 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:49:41.317 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:49:41.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:49:44.390 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:49:44.428 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:49:45.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:51:45.545 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:51:45.734 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:51:46.957 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:51:46.974 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:51:46.974 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:51:46.975 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:51:46.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:51:50.050 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:51:50.089 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:51:51.300 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:53:51.351 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:53:51.393 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:53:52.425 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:53:52.443 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:53:52.443 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:53:52.443 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:53:52.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:53:55.526 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:53:55.564 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:53:56.592 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:55:56.637 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:55:56.772 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:55:58.792 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:55:58.810 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:55:58.810 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:55:58.810 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:55:58.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:56:01.890 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:56:01.941 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:56:03.784 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:58:03.832 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T11:58:03.907 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:58:05.895 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T11:58:05.912 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T11:58:05.912 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T11:58:05.912 INFO:teuthology.misc:Compressing logs... 2021-03-19T11:58:05.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T11:58:08.994 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T11:58:09.033 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T11:58:11.036 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:00:11.078 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:00:11.170 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:00:13.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:00:13.277 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:00:13.278 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:00:13.278 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:00:13.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:00:16.355 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:00:16.392 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:00:17.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:02:17.548 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:02:17.612 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:02:19.653 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:02:19.670 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:02:19.670 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:02:19.670 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:02:19.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:02:22.722 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:02:22.762 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:02:24.651 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:04:24.699 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:04:24.765 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:04:26.707 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:04:26.725 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:04:26.725 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:04:26.725 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:04:26.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:04:29.794 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:04:29.839 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:04:31.891 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:06:31.937 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:06:31.977 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:06:33.838 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:06:33.856 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:06:33.856 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:06:33.856 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:06:33.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:06:36.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:06:36.969 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:06:38.987 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:08:39.046 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:08:39.085 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:08:40.851 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:08:40.868 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:08:40.868 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:08:40.868 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:08:40.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:08:43.938 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:08:43.979 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:08:45.053 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:10:45.097 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:10:45.137 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:10:46.251 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:10:46.268 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:10:46.268 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:10:46.269 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:10:46.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:10:49.350 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:10:49.389 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:10:50.494 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:12:50.539 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:12:50.582 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:12:51.694 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:12:51.711 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:12:51.712 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:12:51.712 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:12:51.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:12:54.790 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:12:54.838 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:12:55.964 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:14:56.011 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:14:56.050 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:14:57.099 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:14:57.115 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:14:57.115 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:14:57.115 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:14:57.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:15:00.194 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:15:00.236 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:15:01.271 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:17:01.315 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:17:01.393 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:17:03.396 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:17:03.414 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:17:03.414 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:17:03.414 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:17:03.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:17:06.466 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:17:06.507 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:17:08.564 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:19:08.609 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:19:08.674 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:19:10.597 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:19:10.620 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:19:10.620 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:19:10.620 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:19:10.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:19:13.698 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:19:13.735 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:19:15.598 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:21:15.642 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:21:15.712 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:21:17.316 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:21:17.346 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:21:17.346 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:21:17.346 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:21:17.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:21:20.422 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:21:20.462 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:21:22.342 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:23:22.389 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:23:22.450 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:23:24.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:23:24.380 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:23:24.380 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:23:24.380 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:23:24.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:23:27.462 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:23:27.506 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:23:29.384 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:25:29.429 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:25:29.501 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:25:31.523 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:25:31.540 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:25:31.540 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:25:31.540 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:25:31.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:25:34.595 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:25:34.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:25:36.659 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:27:36.704 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:27:36.857 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:27:38.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:27:38.705 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:27:38.705 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:27:38.705 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:27:38.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:27:41.762 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:27:41.799 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:27:43.696 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:29:43.741 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:29:43.779 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:29:45.666 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:29:45.684 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:29:45.684 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:29:45.684 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:29:45.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:29:48.742 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:29:48.782 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:29:49.856 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:31:49.900 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:31:49.940 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:31:51.018 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:31:51.036 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:31:51.036 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:31:51.036 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:31:51.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:31:54.114 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:31:54.150 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:31:55.216 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:33:55.263 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:33:55.366 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:33:57.861 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:33:57.883 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:33:57.883 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:33:57.884 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:33:57.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:34:00.962 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:34:01.010 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:34:03.358 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:36:03.404 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:36:03.528 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:36:05.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:36:05.533 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:36:05.534 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:36:05.534 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:36:05.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:36:08.615 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:36:08.654 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:36:10.580 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:38:10.631 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:38:10.673 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:38:11.760 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:38:11.779 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:38:11.779 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:38:11.779 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:38:11.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:38:14.850 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:38:14.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:38:15.961 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:40:16.003 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:40:16.065 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:40:18.044 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:40:18.061 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:40:18.061 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:40:18.061 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:40:18.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:40:21.126 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:40:21.164 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:40:23.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:42:23.059 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:42:23.374 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:42:25.391 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:42:25.412 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:42:25.412 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:42:25.412 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:42:25.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:42:28.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:42:28.536 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:42:30.211 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:44:30.256 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:44:30.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:44:32.271 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:44:32.289 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:44:32.289 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:44:32.290 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:44:32.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:44:35.363 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:44:35.412 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:44:36.729 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:46:36.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:46:36.870 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:46:38.927 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:46:38.945 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:46:38.945 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:46:38.945 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:46:38.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:46:42.018 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:46:42.062 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:46:43.862 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:48:43.911 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:48:43.972 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:48:46.030 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:48:46.054 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:48:46.054 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:48:46.054 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:48:46.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:48:49.123 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:48:49.170 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:48:51.078 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:50:51.122 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:50:51.189 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:50:53.122 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:50:53.139 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:50:53.139 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:50:53.140 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:50:53.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:50:56.198 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:50:56.236 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:50:58.268 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:52:58.315 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:52:58.354 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:52:59.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:52:59.518 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:52:59.519 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:52:59.519 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:52:59.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:53:02.594 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:53:02.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:53:03.704 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:55:03.751 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:55:03.908 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:55:05.818 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:55:05.835 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:55:05.836 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:55:05.836 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:55:05.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:55:08.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:55:08.935 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:55:10.133 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:57:10.878 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:57:10.918 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:57:11.998 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:57:12.015 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:57:12.015 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:57:12.016 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:57:12.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:57:15.074 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:57:15.121 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:57:16.214 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:59:16.284 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T12:59:16.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:59:17.391 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T12:59:17.408 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T12:59:17.408 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T12:59:17.408 INFO:teuthology.misc:Compressing logs... 2021-03-19T12:59:17.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T12:59:20.486 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T12:59:20.524 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T12:59:21.573 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:01:21.618 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:01:21.844 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:01:23.127 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:01:23.150 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:01:23.150 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:01:23.150 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:01:23.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:01:26.210 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:01:26.256 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:01:27.413 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:03:27.461 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:03:27.502 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:03:28.627 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:03:28.647 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:03:28.648 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:03:28.648 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:03:28.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:03:31.718 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:03:31.755 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:03:32.846 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:05:32.892 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:05:33.133 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:05:34.294 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:05:34.311 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:05:34.311 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:05:34.311 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:05:34.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:05:37.378 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:05:37.431 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:05:38.530 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:07:38.578 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:07:38.622 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:07:40.348 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:07:40.366 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:07:40.367 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:07:40.367 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:07:40.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:07:43.431 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:07:43.479 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:07:45.337 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:09:45.388 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:09:45.429 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:09:46.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:09:46.573 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:09:46.574 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:09:46.574 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:09:46.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:09:49.634 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:09:49.672 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:09:50.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:11:50.767 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:11:50.810 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:11:52.804 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:11:52.822 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:11:52.822 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:11:52.822 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:11:52.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:11:55.874 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:11:55.919 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:11:57.907 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:13:57.953 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:13:57.990 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:13:59.059 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:13:59.076 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:13:59.076 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:13:59.077 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:13:59.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:14:02.146 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:14:02.186 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:14:04.026 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:16:04.072 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:16:04.152 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:16:05.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:16:05.224 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:16:05.224 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:16:05.224 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:16:05.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:16:08.294 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:16:08.331 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:16:09.364 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:18:09.408 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:18:09.457 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:18:10.598 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:18:10.617 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:18:10.617 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:18:10.617 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:18:10.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:18:13.698 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:18:13.753 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:18:14.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:20:14.881 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:20:14.924 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:20:15.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:20:15.994 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:20:15.994 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:20:15.994 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:20:15.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:20:19.074 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:20:19.122 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:20:20.208 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:22:20.259 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:22:20.297 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:22:21.400 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:22:21.417 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:22:21.417 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:22:21.418 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:22:21.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:22:24.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:22:24.522 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:22:25.532 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:24:25.579 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:24:25.643 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:24:27.563 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:24:27.581 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:24:27.581 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:24:27.581 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:24:27.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:24:30.658 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:24:30.698 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:24:32.596 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:26:32.654 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:26:32.727 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:26:34.778 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:26:34.803 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:26:34.803 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:26:34.804 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:26:34.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:26:37.862 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:26:37.903 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:26:39.770 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:28:39.817 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:28:39.855 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:28:41.671 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:28:41.689 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:28:41.689 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:28:41.690 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:28:41.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:28:44.774 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:28:44.817 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:28:46.624 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:30:46.671 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:30:46.709 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:30:48.725 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:30:48.744 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:30:48.744 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:30:48.744 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:30:48.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:30:51.810 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:30:51.853 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:30:53.772 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:32:53.822 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:32:53.900 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:32:55.733 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:32:55.751 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:32:55.751 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:32:55.751 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:32:55.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:32:58.818 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:32:58.859 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:33:00.331 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:35:00.382 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:35:00.451 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:35:02.321 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:35:02.339 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:35:02.339 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:35:02.339 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:35:02.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:35:05.410 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:35:05.457 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:35:06.601 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:37:06.649 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:37:06.689 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:37:08.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:37:08.208 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:37:08.208 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:37:08.208 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:37:08.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:37:11.266 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:37:11.305 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:37:12.354 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:39:12.401 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:39:12.440 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:39:13.469 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:39:13.486 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:39:13.486 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:39:13.486 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:39:13.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:39:16.546 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:39:16.583 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:39:17.681 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:41:17.745 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:41:17.784 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:41:18.778 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:41:18.795 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:41:18.795 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:41:18.795 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:41:18.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:41:21.858 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:41:21.895 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:41:22.995 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:43:23.040 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:43:23.086 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:43:24.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:43:24.158 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:43:24.158 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:43:24.159 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:43:24.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:43:27.234 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:43:27.272 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:43:28.419 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:45:28.464 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:45:28.503 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:45:29.627 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:45:29.644 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:45:29.644 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:45:29.644 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:45:29.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:45:32.706 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:45:32.744 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:45:33.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:47:33.839 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:47:33.877 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:47:35.738 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:47:35.756 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:47:35.756 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:47:35.756 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:47:35.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:47:38.818 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:47:38.857 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:47:39.944 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:49:39.990 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:49:40.029 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:49:41.055 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:49:41.072 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:49:41.072 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:49:41.072 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:49:41.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:49:44.130 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:49:44.166 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:49:45.243 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:51:45.291 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:51:45.328 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:51:46.407 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:51:46.425 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:51:46.426 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:51:46.426 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:51:46.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:51:49.506 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:51:49.546 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:51:50.589 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:53:50.641 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:53:50.678 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:53:52.470 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:53:52.497 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:53:52.498 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:53:52.498 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:53:52.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:53:55.558 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:53:55.596 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:53:57.484 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:55:57.528 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:55:57.637 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:55:59.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:55:59.223 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:55:59.223 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:55:59.223 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:55:59.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:56:02.278 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:56:02.317 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:56:03.375 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:58:03.418 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T13:58:03.467 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:58:04.517 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T13:58:04.533 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T13:58:04.533 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T13:58:04.533 INFO:teuthology.misc:Compressing logs... 2021-03-19T13:58:04.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T13:58:07.590 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T13:58:07.629 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T13:58:08.671 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:00:08.716 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:00:08.767 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:00:09.960 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:00:09.981 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:00:09.982 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:00:09.983 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:00:09.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:00:13.058 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:00:13.110 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:00:14.263 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:02:14.309 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:02:14.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:02:15.405 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:02:15.422 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:02:15.422 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:02:15.422 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:02:15.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:02:18.502 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:02:18.538 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:02:19.532 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:04:19.578 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:04:19.616 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:04:20.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:04:20.660 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:04:20.660 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:04:20.660 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:04:20.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:04:23.714 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:04:23.753 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:04:24.774 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:06:24.820 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:06:24.861 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:06:25.943 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:06:25.961 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:06:25.961 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:06:25.961 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:06:25.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:06:29.026 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:06:29.063 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:06:30.139 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:08:30.184 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:08:30.222 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:08:31.246 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:08:31.263 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:08:31.263 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:08:31.264 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:08:31.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:08:34.338 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:08:34.376 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:08:35.555 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:10:35.613 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:10:35.654 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:10:36.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:10:36.874 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:10:36.874 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:10:36.874 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:10:36.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:10:39.939 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:10:39.976 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:10:41.296 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:12:41.352 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:12:41.397 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:12:42.593 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:12:42.612 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:12:42.612 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:12:42.612 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:12:42.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:12:45.666 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:12:45.736 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:12:47.158 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:14:47.203 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:14:47.242 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:14:49.297 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:14:49.332 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:14:49.333 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:14:49.333 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:14:49.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:14:52.386 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:14:52.425 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:14:53.519 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:16:53.566 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:16:53.636 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:16:54.801 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:16:54.818 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:16:54.818 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:16:54.819 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:16:54.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:16:57.890 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:16:57.929 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:16:58.988 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:18:59.035 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:18:59.073 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:19:00.182 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:19:00.199 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:19:00.199 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:19:00.199 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:19:00.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:19:03.266 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:19:03.308 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:19:04.391 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:21:04.607 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:21:04.645 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:21:05.722 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:21:05.739 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:21:05.739 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:21:05.739 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:21:05.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:21:08.802 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:21:08.839 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:21:09.912 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:23:09.957 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:23:09.997 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:23:11.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:23:11.157 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:23:11.157 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:23:11.157 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:23:11.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:23:14.210 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:23:14.273 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:23:15.307 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:25:15.351 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:25:15.390 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:25:16.466 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:25:16.483 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:25:16.483 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:25:16.484 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:25:16.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:25:19.554 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:25:19.590 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:25:20.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:27:20.799 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:27:20.837 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:27:21.915 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:27:21.936 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:27:21.937 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:27:21.937 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:27:21.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:27:24.994 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:27:25.030 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:27:26.070 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:29:26.114 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:29:26.153 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:29:27.215 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:29:27.231 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:29:27.231 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:29:27.231 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:29:27.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:29:30.306 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:29:30.343 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:29:31.430 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:31:31.475 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:31:31.513 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:31:32.529 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:31:32.547 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:31:32.547 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:31:32.547 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:31:32.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:31:35.618 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:31:35.656 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:31:36.701 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:33:36.821 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:33:36.858 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:33:37.883 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:33:37.900 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:33:37.900 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:33:37.900 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:33:37.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:33:40.966 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:33:41.004 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:33:41.987 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:35:42.028 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:35:42.119 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:35:43.202 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:35:43.219 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:35:43.219 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:35:43.219 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:35:43.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:35:46.274 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:35:46.311 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:35:47.384 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:37:47.429 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:37:47.489 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:37:48.650 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:37:48.675 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:37:48.675 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:37:48.676 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:37:48.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:37:51.746 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:37:51.790 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:37:52.921 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:39:52.970 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:39:53.012 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:39:54.016 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:39:54.033 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:39:54.033 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:39:54.033 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:39:54.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:39:57.090 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:39:57.127 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:39:58.177 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:41:58.227 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:41:58.263 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:41:59.336 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:41:59.353 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:41:59.354 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:41:59.354 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:41:59.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:42:02.434 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:42:02.471 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:42:03.528 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:44:03.570 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:44:03.608 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:44:04.656 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:44:04.673 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:44:04.673 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:44:04.674 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:44:04.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:44:07.746 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:44:07.783 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:44:08.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:46:08.862 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:46:08.898 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:46:09.930 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:46:09.947 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:46:09.947 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:46:09.947 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:46:09.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:46:13.027 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:46:13.066 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:46:14.149 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:48:14.249 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:48:14.290 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:48:15.339 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:48:15.357 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:48:15.357 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:48:15.358 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:48:15.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:48:18.435 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:48:18.471 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:48:19.477 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:50:19.519 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:50:19.557 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:50:20.646 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:50:20.663 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:50:20.663 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:50:20.664 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:50:20.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:50:23.714 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:50:23.752 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:50:24.797 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:52:24.849 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:52:24.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:52:25.909 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:52:25.928 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:52:25.928 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:52:25.928 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:52:25.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:52:28.994 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:52:29.033 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:52:30.072 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:54:30.116 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:54:30.156 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:54:31.281 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:54:31.298 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:54:31.299 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:54:31.299 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:54:31.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:54:34.370 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:54:34.409 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:54:35.492 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:56:35.541 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:56:35.582 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:56:36.817 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:56:36.834 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:56:36.835 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:56:36.835 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:56:36.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:56:39.910 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:56:39.947 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:56:41.015 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:58:41.064 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T14:58:41.103 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:58:42.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T14:58:42.209 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T14:58:42.210 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T14:58:42.210 INFO:teuthology.misc:Compressing logs... 2021-03-19T14:58:42.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T14:58:45.286 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T14:58:45.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T14:58:46.373 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:00:46.422 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:00:46.460 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:00:47.547 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:00:47.565 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:00:47.565 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:00:47.565 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:00:47.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:00:49.634 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T15:00:49.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:00:50.744 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:02:50.796 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:02:50.839 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:02:51.905 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:02:51.923 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:02:51.923 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:02:51.923 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:02:51.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:02:52.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:02:52.581 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:02:52.917 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:02:52.917 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:02:52.939 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:02:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T15:02:52.994 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T15:02:53.054 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T15:02:53.054 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:02:53.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T15:02:53.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:02:53.071 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:02:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T15:02:53.128 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T15:02:53.130 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T15:02:53.168 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:02:54.213 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:04:54.260 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:04:54.314 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:04:55.380 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:04:55.397 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:04:55.397 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:04:55.397 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:04:55.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:04:55.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:04:55.628 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:04:58.662 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T15:04:58.700 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:04:59.822 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:06:59.867 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:06:59.923 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:07:01.009 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:07:01.037 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:07:01.037 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:07:01.038 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:07:01.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:07:01.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:07:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:08:01.251 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T15:08:01.292 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:08:02.321 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:10:02.369 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:10:02.415 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:10:03.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:10:03.521 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:10:03.521 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:10:03.521 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:10:03.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:10:03.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:10:03.948 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:10:04.037 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:10:04.038 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:10:04.039 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:10:04.067 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T15:10:04.185 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T15:10:04.185 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:10:04.188 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:10:04.251 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T15:10:04.316 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T15:10:04.316 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:10:04.357 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:10:05.496 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:12:05.561 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:12:05.604 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:12:06.771 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:12:06.790 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:12:06.790 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:12:06.790 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:12:06.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:12:07.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:12:39.414 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:12:39.479 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:12:39.479 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:12:39.481 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:12:39.545 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T15:12:39.623 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T15:12:39.624 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:12:39.625 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:12:39.689 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T15:12:39.755 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T15:12:39.755 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:12:39.799 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:12:40.885 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:14:40.930 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:14:40.968 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:14:43.125 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:14:43.142 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:14:43.143 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:14:43.143 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:14:43.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:14:43.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:14:43.585 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:14:43.652 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:14:43.652 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:14:43.653 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:14:43.822 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:14:43.873 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T15:14:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2021-03-19T15:14:51.079 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2021-03-19T15:14:51.145 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2021-03-19T15:14:53.152 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2021-03-19T15:14:53.980 INFO:teuthology.orchestra.run.smithi060.stderr:tar: ./ceph-osd.3.log: file changed as we read it 2021-03-19T15:14:54.102 INFO:teuthology.orchestra.run.smithi060.stderr:tar: ./ceph-osd.2.log: file changed as we read it 2021-03-19T15:14:55.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: .: file changed as we read it 2021-03-19T15:14:55.861 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T15:14:55.861 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:14:55.940 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:14:57.912 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:16:58.005 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:16:58.044 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:16:59.216 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:16:59.237 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:16:59.237 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:16:59.237 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:16:59.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:16:59.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:16:59.693 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:16:59.762 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:16:59.762 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:16:59.763 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:16:59.832 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T15:16:59.833 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T15:16:59.880 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:17:01.072 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:19:01.120 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:19:01.174 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:19:02.243 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:19:02.263 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:19:02.263 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:19:02.263 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:19:02.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:19:02.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:19:02.772 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:19:02.841 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T15:19:02.841 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T15:19:02.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:19:03.897 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:21:03.950 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:21:03.994 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:21:05.038 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:21:05.056 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:21:05.056 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:21:05.056 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:21:05.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:21:05.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:21:05.591 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:21:05.657 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T15:21:05.659 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T15:21:05.697 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:21:06.737 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:23:06.782 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:23:06.821 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:23:07.860 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:23:07.876 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:23:07.876 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:23:07.877 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:23:07.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:23:08.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:23:08.405 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:23:08.474 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T15:23:08.475 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T15:23:08.512 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:23:09.619 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:25:09.670 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:25:09.713 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:25:10.750 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:25:10.767 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:25:10.768 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:25:10.768 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:25:10.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:25:10.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:25:29.414 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T15:25:29.476 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:25:30.539 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:27:30.584 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:27:30.622 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:27:31.748 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:27:31.765 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:27:31.765 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:27:31.766 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:27:31.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:27:50.210 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T15:27:50.250 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:27:51.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:29:51.416 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:29:51.458 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:29:52.557 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:29:52.574 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:29:52.574 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:29:52.575 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:29:52.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:29:52.966 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T15:29:53.002 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:29:54.074 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:31:54.119 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:31:54.161 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:31:55.236 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:31:55.254 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:31:55.254 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:31:55.255 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:31:55.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:31:55.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:31:55.562 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:31:55.764 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:31:55.845 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:31:55.846 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:31:55.890 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:31:55.890 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:31:55.891 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:31:55.917 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T15:31:55.957 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T15:31:55.984 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T15:31:55.984 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:31:56.022 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T15:31:56.022 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:31:56.024 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:31:56.050 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T15:31:56.088 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T15:31:56.117 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T15:31:56.117 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:31:56.154 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T15:31:56.154 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:31:56.197 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:31:57.308 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:33:57.366 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:33:57.406 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:33:58.452 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:33:58.470 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:33:58.470 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:33:58.470 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:33:58.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:33:58.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:33:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:34:01.762 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T15:34:01.799 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:34:02.866 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:36:02.909 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:36:02.948 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:36:04.030 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:36:04.050 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:36:04.050 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:36:04.050 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:36:04.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:36:04.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:36:04.288 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:36:04.485 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:36:04.524 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:36:04.524 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:36:04.561 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:36:04.561 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:36:04.562 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:36:04.713 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:36:04.739 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T15:36:04.768 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T15:36:04.806 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T15:36:04.806 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:36:04.833 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T15:36:04.833 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:36:04.874 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:36:05.975 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:38:06.023 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:38:06.069 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:38:07.210 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:38:07.229 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:38:07.230 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:38:07.230 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:38:07.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:38:07.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:38:07.471 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:38:07.671 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:38:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:38:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:38:07.740 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:38:07.740 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:38:07.741 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:38:07.807 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T15:38:07.808 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T15:38:07.845 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:38:08.941 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:40:08.984 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:40:09.024 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:40:10.094 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:40:10.110 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:40:10.110 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:40:10.111 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:40:10.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:40:10.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:40:10.345 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:40:10.557 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:40:10.585 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:40:10.585 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:40:10.624 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:40:10.624 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:40:10.625 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:40:10.652 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T15:40:10.668 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T15:40:10.706 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:40:11.806 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:42:11.852 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:42:11.895 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:42:12.911 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:42:12.928 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:42:12.929 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:42:12.929 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:42:12.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:42:13.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:42:13.165 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:42:13.398 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:42:13.399 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:42:13.439 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:42:13.468 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T15:42:13.481 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T15:42:13.518 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:42:14.567 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:44:14.612 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:44:14.652 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:44:15.709 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:44:15.726 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:44:15.727 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:44:15.727 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:44:15.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:44:15.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:44:15.972 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:44:16.212 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:44:16.212 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:44:16.252 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:44:16.283 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T15:44:16.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T15:44:16.334 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:44:17.443 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:46:17.488 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:46:17.526 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:46:18.558 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:46:18.575 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:46:18.575 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:46:18.576 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:46:18.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:46:18.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:46:18.821 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:46:19.061 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:46:19.061 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:46:19.101 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:46:19.131 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T15:46:19.143 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T15:46:19.180 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:46:20.204 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:48:20.248 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:48:20.289 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:48:21.455 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:48:21.475 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:48:21.475 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:48:21.475 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:48:21.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:48:21.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:48:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T15:48:21.957 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T15:48:21.957 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T15:48:21.998 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:48:22.026 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T15:48:22.040 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T15:48:22.078 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:48:23.250 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:50:23.297 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:50:23.336 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:50:24.412 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:50:24.430 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:50:24.431 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:50:24.431 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:50:24.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:50:42.887 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T15:50:42.977 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:50:44.057 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:52:44.107 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:52:44.144 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:52:45.187 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:52:45.205 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:52:45.205 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:52:45.205 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:52:45.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:52:45.307 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:52:46.404 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:52:47.500 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:52:48.601 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:52:49.698 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:52:50.793 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:52:51.892 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:52:52.950 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:52:54.049 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:52:55.105 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:52:55.106 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T15:52:55.142 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:52:56.153 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:54:56.202 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:54:56.242 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:54:57.303 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:54:57.320 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:54:57.320 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:54:57.320 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:54:57.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:54:57.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:54:57.689 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T15:55:00.738 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T15:55:00.776 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:55:01.839 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:57:01.886 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:57:01.924 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:57:02.951 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:57:02.967 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:57:02.967 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:57:02.967 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:57:02.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:57:03.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:57:03.317 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T15:57:03.358 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:57:04.455 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:57:05.512 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:57:06.610 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:57:07.710 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:57:08.769 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:57:09.865 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:57:10.925 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:57:12.023 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:57:13.081 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T15:57:13.081 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T15:57:13.120 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:57:14.136 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:59:14.182 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T15:59:14.219 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T15:59:15.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T15:59:15.292 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T15:59:15.292 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T15:59:15.292 INFO:teuthology.misc:Compressing logs... 2021-03-19T15:59:15.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:59:15.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T15:59:15.616 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:00:15.608 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T16:00:15.649 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:00:16.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:02:16.872 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:02:16.921 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:02:18.037 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:02:18.053 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:02:18.054 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:02:18.054 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:02:18.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:02:18.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:02:18.515 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:02:51.603 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:02:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:02:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:02:51.635 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:02:51.635 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:02:51.637 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:02:51.692 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T16:02:51.782 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T16:02:51.782 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:02:51.784 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:02:51.840 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T16:02:51.921 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T16:02:51.921 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:02:51.959 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:02:53.033 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:04:53.082 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:04:53.125 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:04:54.241 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:04:54.260 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:04:54.260 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:04:54.260 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:04:54.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:04:54.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:04:54.601 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:04:54.891 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:04:54.908 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:04:54.908 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:04:54.949 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:04:54.949 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:04:54.950 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:04:55.092 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:04:55.109 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T16:04:55.109 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/7031cf1a-88cc-11eb-909e-001a4aab830c/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-19T16:04:55.129 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:04:55.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:04:56.301 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:06:56.349 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:06:56.390 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:06:57.444 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:06:57.463 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:06:57.463 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:06:57.463 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:06:57.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:06:57.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:06:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:06:58.147 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:06:58.147 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:06:58.196 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:06:58.330 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:06:58.349 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T16:06:58.349 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/7031cf1a-88cc-11eb-909e-001a4aab830c/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-19T16:06:58.392 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/7031cf1a-88cc-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T16:06:58.409 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:06:58.449 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:06:59.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:08:59.548 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:08:59.585 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:09:00.677 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:09:00.696 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:09:00.697 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:09:00.697 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:09:00.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:09:01.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:09:01.035 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:09:01.381 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:09:01.381 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:09:01.423 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:09:01.566 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:09:01.589 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T16:09:01.590 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/7031cf1a-88cc-11eb-909e-001a4aab830c/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-19T16:09:01.590 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/7031cf1a-88cc-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T16:09:01.596 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:09:01.638 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:09:02.735 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:11:02.781 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:11:02.821 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:11:03.920 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:11:03.937 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:11:03.938 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:11:03.938 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:11:03.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:11:04.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:11:04.275 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:11:04.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:11:04.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:11:04.655 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:11:04.800 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:11:04.818 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T16:11:04.818 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/7031cf1a-88cc-11eb-909e-001a4aab830c/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-19T16:11:04.818 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/7031cf1a-88cc-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T16:11:04.838 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:11:04.877 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:11:05.927 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:13:05.973 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:13:06.015 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:13:07.152 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:13:07.169 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:13:07.169 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:13:07.169 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:13:07.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:13:07.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:13:07.500 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:13:07.825 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:13:07.825 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:13:07.867 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:13:08.007 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:13:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T16:13:08.033 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:13:08.075 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:13:09.205 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:15:09.252 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:15:09.292 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:15:10.344 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:15:10.362 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:15:10.362 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:15:10.362 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:15:10.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:15:10.424 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T16:15:11.521 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T16:15:12.619 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T16:15:13.678 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T16:15:14.775 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T16:15:15.834 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T16:15:16.932 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T16:15:18.031 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T16:15:19.093 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T16:15:20.189 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T16:15:20.190 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T16:15:20.227 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:15:21.320 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:17:21.366 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:17:21.405 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:17:22.482 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:17:22.501 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:17:22.501 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:17:22.502 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:17:22.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:17:30.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:17:30.594 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:17:33.574 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T16:17:33.627 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:17:34.697 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:19:34.747 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:19:34.784 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:19:35.828 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:19:35.845 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:19:35.845 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:19:35.846 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:19:35.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:19:36.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:19:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:19:36.583 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:19:36.653 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:19:36.653 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:19:36.673 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:19:36.673 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:19:36.675 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:19:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T16:19:36.731 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T16:19:36.774 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T16:19:36.774 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:19:36.791 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T16:19:36.792 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:19:36.794 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:19:36.843 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T16:19:36.850 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T16:19:36.912 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T16:19:36.912 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:19:36.929 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T16:19:36.929 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:19:36.968 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:19:38.028 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:21:38.074 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:21:38.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:21:39.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:21:39.156 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:21:39.157 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:21:39.157 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:21:39.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:21:39.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:21:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:22:39.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T16:22:39.938 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:22:40.988 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:24:41.034 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:24:41.094 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:24:42.173 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:24:42.190 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:24:42.190 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:24:42.191 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:24:42.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:24:42.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:24:42.436 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:24:42.986 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:24:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:24:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:24:43.039 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:24:43.040 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:24:43.042 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:24:43.223 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:24:43.251 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T16:24:43.262 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T16:24:43.319 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T16:24:43.319 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:24:43.335 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T16:24:43.335 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:24:43.379 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:24:44.510 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:26:44.556 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:26:44.596 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:26:45.684 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:26:45.701 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:26:45.701 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:26:45.701 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:26:45.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:26:45.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:26:45.939 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:26:46.239 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:26:46.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:26:46.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:26:46.298 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:26:46.298 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:26:46.300 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:26:46.485 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:26:46.512 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T16:26:46.514 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T16:26:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T16:26:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:26:46.595 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T16:26:46.595 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:26:46.673 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:26:47.686 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:28:47.732 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:28:47.773 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:28:48.808 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:28:48.826 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:28:48.826 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:28:48.826 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:28:48.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:28:49.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:28:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:28:49.337 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:28:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:28:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:28:49.396 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:28:49.397 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:28:49.398 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:28:49.738 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:28:49.894 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:28:50.912 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:30:50.954 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:30:50.992 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:30:52.035 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:30:52.051 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:30:52.052 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:30:52.052 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:30:52.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:30:52.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:30:52.307 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:30:52.619 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:30:52.654 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:30:52.654 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:30:52.679 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:30:52.679 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:30:52.681 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:30:53.032 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:30:53.059 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T16:30:53.060 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/05e9f91c-88d0-11eb-909e-001a4aab830c/ceph-mgr.smithi060.ewhthu.log.gz already exists; not overwritten 2021-03-19T16:30:53.076 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:30:53.118 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:30:54.202 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:32:54.248 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:32:54.292 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:32:55.591 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:32:55.608 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:32:55.608 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:32:55.608 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:32:55.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:32:55.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:32:55.906 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:32:56.216 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:32:56.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:32:56.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:32:56.288 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:32:56.289 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:32:56.291 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:32:56.825 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:32:56.858 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T16:32:56.882 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/05e9f91c-88d0-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T16:32:56.886 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T16:32:56.894 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/05e9f91c-88d0-11eb-909e-001a4aab830c/ceph-mgr.smithi060.ewhthu.log.gz already exists; not overwritten 2021-03-19T16:32:56.909 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:32:56.949 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:32:57.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:34:58.041 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:34:58.078 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:34:59.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:34:59.211 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:34:59.211 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:34:59.212 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:34:59.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:34:59.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:34:59.479 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:34:59.760 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:34:59.794 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:34:59.794 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:34:59.820 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:34:59.820 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:34:59.825 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:35:00.170 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:35:00.198 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T16:35:00.198 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/05e9f91c-88d0-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T16:35:00.198 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/05e9f91c-88d0-11eb-909e-001a4aab830c/ceph-mgr.smithi060.ewhthu.log.gz already exists; not overwritten 2021-03-19T16:35:00.207 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:35:00.247 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:35:02.289 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:37:02.337 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:37:02.379 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:37:04.255 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:37:04.272 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:37:04.272 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:37:04.273 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:37:04.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:37:04.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:37:04.524 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:37:04.830 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:37:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:37:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:37:04.888 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:37:04.888 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:37:04.890 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:37:05.244 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:37:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T16:37:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/05e9f91c-88d0-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T16:37:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/05e9f91c-88d0-11eb-909e-001a4aab830c/ceph-mgr.smithi060.ewhthu.log.gz already exists; not overwritten 2021-03-19T16:37:05.285 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:37:05.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:37:06.374 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:39:06.421 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:39:06.465 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:39:07.582 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:39:07.599 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:39:07.599 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:39:07.599 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:39:07.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:39:26.147 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T16:39:26.187 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:39:27.243 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:41:27.289 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:41:27.329 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:41:28.383 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:41:28.400 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:41:28.400 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:41:28.400 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:41:28.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:41:31.458 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T16:41:31.498 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:41:32.543 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:43:32.590 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:43:32.629 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:43:33.703 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:43:33.721 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:43:33.721 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:43:33.721 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:43:33.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:43:34.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:43:34.063 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:43:37.126 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T16:43:37.163 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:43:38.230 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:45:38.276 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:45:38.314 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:45:39.394 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:45:39.412 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:45:39.412 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:45:39.412 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:45:39.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:46:39.416 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T16:46:39.456 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:46:40.565 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:48:40.616 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:48:40.655 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:48:41.762 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:48:41.778 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:48:41.779 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:48:41.779 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:48:41.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:48:42.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:48:42.097 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T16:48:42.310 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:48:42.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:48:42.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:48:42.395 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:48:42.395 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:48:42.396 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:48:42.544 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:48:42.559 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T16:48:42.593 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T16:48:42.616 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T16:48:42.616 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:48:42.659 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T16:48:42.659 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:48:42.698 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:48:43.810 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:50:43.855 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:50:43.894 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:50:44.963 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:50:44.980 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:50:44.980 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:50:44.980 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:50:44.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:50:45.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:50:45.540 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:50:45.870 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:50:45.870 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:50:45.872 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:50:46.023 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:50:46.069 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T16:50:46.135 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T16:50:46.135 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:50:46.175 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:50:47.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:52:47.254 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:52:47.292 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:52:48.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:52:48.352 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:52:48.352 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:52:48.352 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:52:48.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:52:48.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:52:48.928 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T16:52:48.993 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T16:52:48.993 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T16:52:48.995 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:52:49.059 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T16:52:49.060 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:52:49.099 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:52:50.165 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:54:50.209 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:54:50.249 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:54:51.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:54:51.370 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:54:51.370 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:54:51.370 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:54:51.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:54:51.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:54:51.986 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:54:52.051 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T16:54:52.052 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:54:52.090 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:54:53.217 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:56:53.263 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:56:53.302 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:56:54.370 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:56:54.387 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:56:54.387 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:56:54.387 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:56:54.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:56:54.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:56:55.012 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:56:55.079 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T16:56:55.081 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:56:55.119 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:56:56.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:58:56.281 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T16:58:56.333 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:58:57.510 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T16:58:57.528 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T16:58:57.528 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T16:58:57.528 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:58:57.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:58:57.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T16:58:58.170 INFO:teuthology.misc:Compressing logs... 2021-03-19T16:58:58.244 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T16:58:58.245 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T16:58:58.288 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T16:58:59.464 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:00:59.519 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:00:59.602 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:01:01.038 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:01:01.058 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:01:01.058 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:01:01.058 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:01:01.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:01:01.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:01:01.756 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:01:01.839 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:01:01.840 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:01:01.879 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:01:03.853 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:03:03.904 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:03:03.947 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:03:05.004 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:03:05.020 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:03:05.021 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:03:05.021 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:03:05.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:03:05.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:03:05.686 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:03:05.762 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:03:05.763 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:03:05.863 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:03:06.915 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:05:06.971 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:05:07.200 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:05:09.038 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:05:09.058 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:05:09.058 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:05:09.058 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:05:09.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:05:09.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:05:09.711 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:05:09.776 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:05:09.778 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:05:09.814 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:05:10.863 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:07:10.920 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:07:11.011 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:07:13.411 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:07:13.440 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:07:13.440 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:07:13.441 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:07:13.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:07:13.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:07:14.105 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:07:14.175 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:07:14.177 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:07:14.224 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:07:16.147 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:09:16.201 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:09:16.270 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:09:18.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:09:18.835 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:09:18.836 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:09:18.836 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:09:18.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:09:19.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:09:19.648 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:09:19.687 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:09:19.689 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:09:19.727 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:09:21.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:11:21.594 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:11:21.675 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:11:23.638 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:11:23.655 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:11:23.656 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:11:23.656 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:11:23.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:11:24.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:11:24.394 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:11:24.410 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T17:11:24.486 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:11:24.488 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:11:24.526 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:11:25.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:13:25.897 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:13:26.084 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:13:55.294 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:13:55.485 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:13:55.518 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:13:55.524 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:13:55.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:14:28.671 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:14:29.765 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:14:30.880 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:14:31.987 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:14:33.190 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:14:34.293 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:14:35.355 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:14:36.455 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:14:37.561 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:14:38.622 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:14:38.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T17:14:38.812 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:14:41.734 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:16:42.249 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:16:42.354 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:16:44.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:16:44.200 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:16:44.200 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:16:44.200 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:16:44.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:16:45.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:16:45.217 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:16:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:16:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:16:45.818 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:16:45.875 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:16:45.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:16:45.907 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:16:45.969 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:16:47.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:18:47.077 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:18:47.137 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:18:49.027 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:18:49.044 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:18:49.044 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:18:49.044 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:18:49.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:18:49.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:18:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:18:49.533 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:18:49.533 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:18:49.569 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:18:49.605 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:18:49.632 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:18:49.634 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:18:49.675 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:18:50.792 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:20:50.840 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:20:50.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:20:52.704 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:20:52.721 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:20:52.721 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:20:52.721 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:20:52.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:20:52.939 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:20:52.969 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:20:53.211 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:20:53.211 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:20:53.249 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:20:53.282 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:20:53.314 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:20:53.316 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:20:53.355 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:20:54.447 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:22:54.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:22:54.565 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:22:55.854 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:22:55.871 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:22:55.871 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:22:55.872 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:22:55.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:22:56.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:22:56.132 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:22:56.372 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:22:56.372 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:22:56.402 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:22:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:22:56.466 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:22:56.468 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:22:56.507 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:22:57.603 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:24:57.656 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:24:57.936 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:24:59.139 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:24:59.156 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:24:59.156 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:24:59.156 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:24:59.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:24:59.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:24:59.405 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:24:59.643 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:24:59.643 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:24:59.683 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:24:59.713 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:24:59.748 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:24:59.750 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:24:59.788 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:25:00.840 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:27:00.903 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:27:00.991 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:27:02.213 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:27:02.230 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:27:02.230 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:27:02.230 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:27:02.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:27:02.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:27:02.471 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:27:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:27:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:27:02.737 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:27:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:27:02.802 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:27:02.803 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:27:02.838 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:27:03.900 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:29:03.949 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:29:04.116 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:29:06.001 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:29:06.020 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:29:06.020 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:29:06.020 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:29:06.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:29:06.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:29:06.283 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:29:09.316 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T17:29:09.356 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:29:10.546 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:31:10.595 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:31:10.832 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:31:11.912 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:31:11.929 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:31:11.930 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:31:11.930 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:31:11.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:31:12.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:31:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:32:12.146 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T17:32:12.256 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:32:14.096 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:34:14.148 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:34:14.187 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:34:15.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:34:15.256 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:34:15.256 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:34:15.256 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:34:15.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:34:15.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:34:15.522 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:34:15.700 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:34:15.734 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:34:15.734 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:34:15.775 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:34:15.775 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:34:15.776 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:34:15.805 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:34:15.875 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T17:34:15.876 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:34:15.943 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:34:15.971 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T17:34:15.982 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T17:34:16.041 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T17:34:16.041 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:34:16.066 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T17:34:16.067 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:34:16.104 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:34:17.203 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:36:17.252 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:36:17.293 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:36:18.329 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:36:18.346 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:36:18.346 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:36:18.346 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:36:18.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:36:18.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:36:18.588 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:36:18.791 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:36:18.820 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:36:18.820 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:36:18.856 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:36:18.857 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:36:18.859 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:36:18.890 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:36:18.959 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T17:36:18.960 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:36:19.019 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:36:19.046 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T17:36:19.056 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T17:36:19.116 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T17:36:19.116 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:36:19.141 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T17:36:19.141 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:36:19.178 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:36:20.248 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:38:20.306 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:38:20.347 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:38:21.442 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:38:21.461 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:38:21.462 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:38:21.462 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:38:21.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:38:21.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:38:21.711 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:38:21.908 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:38:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:38:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:38:21.975 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:38:21.975 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:38:21.976 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:38:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:38:22.040 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:38:22.041 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:38:22.079 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:38:23.237 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:40:23.284 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:40:25.966 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:40:27.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:40:27.157 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:40:27.157 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:40:27.157 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:40:27.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:40:27.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:40:27.417 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:40:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:40:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:40:27.675 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:40:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:40:27.740 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:40:27.741 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:40:27.784 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:40:28.924 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:42:28.967 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:42:29.009 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:42:30.997 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:42:31.017 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:42:31.017 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:42:31.017 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:42:31.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:42:31.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:42:31.261 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:42:31.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:42:31.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:42:31.535 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:42:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:42:31.600 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:42:31.602 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:42:31.651 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:42:33.448 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:44:33.504 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:44:33.542 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:44:35.449 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:44:35.480 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:44:35.481 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:44:35.481 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:44:35.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:44:35.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:44:35.729 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:44:35.958 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:44:35.959 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:44:35.999 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:44:36.032 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:44:36.065 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:44:36.066 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:44:36.105 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:44:37.288 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:46:37.333 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:46:37.392 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:46:38.481 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:46:38.503 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:46:38.504 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:46:38.504 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:46:38.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:46:38.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:46:38.767 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:47:13.223 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T17:47:13.262 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:47:14.918 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:49:14.966 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:49:15.003 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:49:16.042 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:49:16.060 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:49:16.060 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:49:16.060 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:49:16.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:49:16.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:49:16.306 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:49:16.333 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:49:17.391 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:49:18.492 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:49:19.591 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:49:20.688 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:49:21.747 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:49:22.844 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:49:23.903 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:49:25.001 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:49:26.102 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T17:49:26.103 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T17:49:26.148 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:49:27.257 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:51:27.305 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:51:27.344 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:51:28.507 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:51:28.527 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:51:28.527 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:51:28.528 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:51:28.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:51:28.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:51:28.769 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:51:36.620 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:51:36.664 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:51:36.665 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:51:36.701 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:51:36.701 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:51:36.702 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:51:36.737 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:51:36.766 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:51:36.809 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T17:51:36.810 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:51:36.837 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T17:51:36.837 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:51:36.839 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:51:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T17:51:36.903 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T17:51:36.948 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T17:51:36.948 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:51:36.972 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T17:51:36.973 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:51:37.012 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:51:38.096 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:53:38.145 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:53:38.212 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:53:40.465 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:53:40.482 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:53:40.482 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:53:40.482 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:53:40.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:53:40.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:53:40.757 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:53:58.691 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T17:53:58.759 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:54:00.438 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:56:00.491 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:56:00.648 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:56:02.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:56:02.220 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:56:02.221 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:56:02.221 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:56:02.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:56:02.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:56:02.468 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:56:02.685 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:56:02.719 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:56:02.719 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:56:02.768 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:56:02.768 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:56:02.769 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:56:02.797 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:56:02.865 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T17:56:02.866 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:56:02.919 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:56:02.946 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T17:56:02.958 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T17:56:03.015 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T17:56:03.016 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:56:03.041 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T17:56:03.042 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:56:03.085 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:56:04.257 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:56:04.305 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/ Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 498, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 463, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 320, in report_job response.raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/sage-2021-03-18_16:32:54-rados:cephadm:dashboard-cephadm-iscsi-test-distro-basic-smithi/jobs/5977701/ 2021-03-19T17:58:04.307 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T17:58:04.350 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:58:05.533 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T17:58:05.551 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T17:58:05.551 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T17:58:05.552 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:58:05.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:58:05.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T17:58:05.795 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:58:05.999 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T17:58:06.032 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:58:06.032 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:58:06.071 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T17:58:06.071 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T17:58:06.072 INFO:teuthology.misc:Compressing logs... 2021-03-19T17:58:06.103 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T17:58:06.137 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T17:58:06.139 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T17:58:06.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T17:58:07.351 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:00:07.478 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:00:07.518 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:00:08.596 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:00:08.614 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:00:08.614 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:00:08.614 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:00:08.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:00:08.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:00:08.881 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:00:09.089 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:00:09.130 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:00:09.130 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:00:09.164 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:00:09.165 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:00:09.167 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:00:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:00:09.240 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:00:09.242 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:00:09.287 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:00:10.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:02:10.549 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:02:10.591 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:02:13.080 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:02:13.109 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:02:13.109 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:02:13.109 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:02:13.109 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:02:13.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:02:13.394 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:02:13.622 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:02:13.622 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:02:13.658 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:02:13.689 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:02:13.724 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:02:13.726 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:02:13.787 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:02:15.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:04:15.072 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:04:15.122 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:04:16.372 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:04:16.389 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:04:16.390 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:04:16.390 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:04:16.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:04:16.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:04:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:04:17.025 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:04:17.025 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:04:17.133 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:04:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:04:17.208 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:04:17.209 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:04:17.255 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:04:19.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:06:19.234 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:06:19.317 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:06:21.031 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:06:21.048 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:06:21.048 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:06:21.048 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:06:21.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:06:21.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:06:21.306 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:06:21.900 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:06:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:06:22.098 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:06:22.128 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:06:22.278 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:06:22.281 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:06:22.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:06:24.199 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:08:24.255 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:08:24.327 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:08:26.422 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:08:26.441 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:08:26.441 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:08:26.441 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:08:26.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:08:26.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:08:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:08:27.721 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:08:27.721 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:08:28.175 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:08:28.205 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:08:28.401 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:08:28.484 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:08:28.523 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:08:30.365 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:10:30.413 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:10:30.583 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:10:31.735 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:10:31.752 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:10:31.752 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:10:31.753 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:10:31.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:10:31.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:10:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:10:32.891 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:10:32.891 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:10:33.388 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:10:33.419 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:10:33.576 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:10:33.578 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:10:33.618 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:10:34.713 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:12:34.764 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:12:34.803 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:12:35.899 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:12:35.926 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:12:35.926 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:12:35.927 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:12:35.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:12:36.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:12:36.191 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:12:36.959 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:12:36.960 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:12:37.141 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:12:37.171 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:12:37.293 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:12:37.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:12:37.332 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:12:38.445 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:14:38.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:14:38.543 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:14:39.765 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:14:39.783 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:14:39.784 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:14:39.784 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:14:39.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:14:40.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:14:40.032 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:14:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:14:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:14:40.730 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:14:40.760 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:14:40.902 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:14:40.905 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:14:41.000 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:14:42.064 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:16:42.111 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:16:42.150 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:16:43.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:16:44.013 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:16:44.013 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:16:44.014 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:16:44.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:16:44.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:16:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:16:44.525 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:16:44.525 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:16:44.573 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:16:44.602 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:16:44.648 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:16:44.649 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:16:44.689 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:16:45.970 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:18:46.022 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:18:46.062 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:18:48.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:18:48.382 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:18:48.382 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:18:48.382 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:18:48.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:18:48.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:18:48.665 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:18:48.915 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:18:48.915 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:18:48.987 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:18:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:18:49.020 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:18:49.022 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:18:49.096 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:18:51.591 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:20:51.642 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:20:52.233 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:20:54.374 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:20:54.398 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:20:54.398 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:20:54.398 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:20:54.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:20:54.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:20:54.641 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:20:54.931 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:20:54.931 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:20:54.984 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:20:55.016 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:20:55.059 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:20:55.060 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:20:55.102 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:20:57.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:22:57.429 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:22:57.504 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:22:59.853 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:22:59.872 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:22:59.872 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:22:59.872 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:22:59.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:23:00.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:23:00.130 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:23:01.373 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:23:01.373 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:23:01.553 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:23:01.583 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:23:01.589 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:23:01.592 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:23:01.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:23:03.735 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:25:03.796 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:25:03.885 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:25:05.062 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:25:05.100 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:25:05.100 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:25:05.100 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:25:05.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:25:05.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:25:05.361 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:25:05.590 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:25:05.590 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:25:05.627 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:25:05.659 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:25:05.736 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:25:05.738 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:25:05.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:25:06.854 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:27:06.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:27:06.995 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:27:08.834 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:27:08.855 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:27:08.855 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:27:08.855 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:27:08.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:27:09.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:27:09.098 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:27:09.416 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:27:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:27:09.534 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:27:09.576 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:27:09.600 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:27:09.601 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:27:09.641 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:27:10.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:29:10.939 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:29:10.981 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:29:12.080 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:29:12.106 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:29:12.107 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:29:12.107 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:29:12.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:29:12.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:29:12.359 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:29:12.654 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:29:12.654 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:29:12.690 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:29:12.724 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:29:12.764 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:29:12.765 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:29:12.808 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:29:13.960 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:31:14.014 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:31:14.071 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:31:15.273 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:31:15.291 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:31:15.292 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:31:15.292 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:31:15.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:31:15.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:31:15.558 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:31:15.783 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:31:15.783 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:31:15.821 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:31:15.852 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:31:15.887 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:31:15.888 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:31:15.935 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:31:17.408 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:33:17.457 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:33:17.508 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:33:18.619 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:33:18.636 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:33:18.636 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:33:18.637 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:33:18.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:33:18.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:33:18.885 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:33:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:33:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:33:19.206 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:33:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:33:19.273 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:33:19.275 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:33:19.347 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:33:20.535 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:35:20.583 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:35:20.622 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:35:21.749 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:35:21.766 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:35:21.766 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:35:21.767 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:35:21.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:35:21.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:35:22.018 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:35:22.279 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:35:22.279 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:35:22.317 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:35:22.351 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:35:22.382 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:35:22.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:35:22.423 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:35:23.475 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:37:23.528 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:37:23.572 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:37:24.596 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:37:24.613 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:37:24.613 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:37:24.613 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:37:24.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:37:24.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:37:24.878 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:37:25.088 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:37:25.088 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:37:25.127 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:37:25.156 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:37:25.193 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:37:25.194 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:37:25.231 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:37:26.314 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:39:26.362 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:39:26.400 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:39:27.482 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:39:27.502 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:39:27.503 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:39:27.503 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:39:27.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:39:27.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:39:27.748 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:39:27.990 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:39:27.990 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:39:28.030 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:39:28.057 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:39:28.094 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:39:28.095 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:39:28.137 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:39:29.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:41:29.294 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:41:29.333 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:41:30.441 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:41:30.458 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:41:30.458 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:41:30.458 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:41:30.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:41:30.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:41:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:41:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:41:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:41:30.971 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:41:30.998 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:41:31.035 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:41:31.036 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:41:31.074 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:41:32.122 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:43:32.166 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:43:32.206 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:43:33.245 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:43:33.262 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:43:33.263 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:43:33.263 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:43:33.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:43:33.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:43:33.520 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:43:34.121 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:43:34.122 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:43:34.162 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:43:34.192 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:43:34.229 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:43:34.231 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:43:34.269 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:43:35.308 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:45:35.352 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:45:35.390 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:45:36.413 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:45:36.430 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:45:36.430 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:45:36.430 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:45:36.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:45:36.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:45:36.672 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:45:36.936 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:45:36.937 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:45:36.966 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:45:37.008 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:45:37.032 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:45:37.034 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:45:37.076 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:45:38.207 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:47:38.254 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:47:38.293 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:47:39.391 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:47:39.410 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:47:39.411 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:47:39.411 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:47:39.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:47:39.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:47:39.653 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:47:39.884 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:47:39.884 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:47:39.923 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:47:39.953 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:47:39.988 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:47:39.990 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:47:40.027 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:47:41.107 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:49:41.155 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:49:41.200 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:49:42.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:49:42.309 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:49:42.309 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:49:42.310 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:49:42.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:49:42.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:49:42.567 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:49:42.781 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:49:42.781 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:49:42.821 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:49:42.849 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:49:42.887 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T18:49:42.888 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T18:49:42.925 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:49:43.973 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:51:44.017 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:51:44.055 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:51:45.136 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:51:45.153 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:51:45.153 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:51:45.154 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:51:45.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:51:45.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:51:45.401 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:51:48.450 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T18:51:48.508 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:51:49.617 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:53:49.669 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:53:49.712 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:53:50.746 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:53:50.764 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:53:50.764 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:53:50.764 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:53:50.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:53:50.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:53:51.008 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:53:54.050 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T18:53:54.088 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:53:55.186 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:55:55.232 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:55:55.276 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:55:56.382 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:55:56.399 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:55:56.400 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:55:56.400 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:55:56.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:55:56.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:55:56.658 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:56:06.850 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T18:56:06.888 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:56:07.999 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:58:08.051 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T18:58:08.100 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:58:09.287 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T18:58:09.305 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T18:58:09.306 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T18:58:09.306 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:58:09.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:58:09.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T18:58:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:58:09.744 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T18:58:09.775 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:58:09.775 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:58:09.853 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T18:58:09.853 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:58:09.854 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:58:09.881 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:58:09.882 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T18:58:09.951 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T18:58:09.951 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:58:09.975 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T18:58:09.976 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:58:09.977 INFO:teuthology.misc:Compressing logs... 2021-03-19T18:58:10.023 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T18:58:10.042 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T18:58:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T18:58:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:58:10.115 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T18:58:10.115 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T18:58:10.153 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T18:58:11.222 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:00:11.276 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:00:11.322 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:00:12.611 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:00:12.630 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:00:12.630 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:00:12.630 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:00:12.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:00:12.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:00:12.873 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:00:15.910 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T19:00:15.985 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:00:17.228 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:02:17.274 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:02:17.318 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:02:18.446 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:02:18.465 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:02:18.465 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:02:18.465 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:02:18.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:02:18.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:02:18.728 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:02:18.927 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:02:18.959 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:02:18.959 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:02:19.028 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:02:19.028 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:02:19.029 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:02:19.059 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:02:19.126 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:02:19.126 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:02:19.173 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:02:19.202 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:02:19.222 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:02:19.271 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:02:19.271 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:02:19.296 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:02:19.296 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:02:19.338 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:02:20.433 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:04:20.484 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:04:20.541 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:04:21.666 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:04:21.683 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:04:21.683 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:04:21.683 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:04:21.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:04:21.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:04:21.932 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:04:22.143 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:04:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:04:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:04:22.211 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:04:22.211 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:04:22.212 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:04:22.245 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:04:22.278 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T19:04:22.280 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:04:22.320 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:04:23.402 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:06:23.454 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:06:23.501 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:06:24.632 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:06:24.650 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:06:24.650 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:06:24.650 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:06:24.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:06:24.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:06:24.894 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:06:25.148 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:06:25.148 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:06:25.175 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:06:25.216 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:06:25.240 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T19:06:25.241 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:06:25.279 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:06:26.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:08:26.394 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:08:26.434 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:08:27.530 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:08:27.546 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:08:27.546 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:08:27.547 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:08:27.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:08:27.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:08:27.809 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:08:28.038 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:08:28.038 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:08:28.078 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:08:28.109 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:08:28.145 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T19:08:28.146 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:08:28.186 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:08:29.314 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:10:29.362 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:10:29.400 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:10:30.471 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:10:30.488 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:10:30.488 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:10:30.488 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:10:30.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:10:30.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:10:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:10:30.958 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:10:30.959 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:10:30.996 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:10:31.029 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:10:31.066 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T19:10:31.069 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:10:31.106 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:10:32.158 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:12:32.205 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:12:32.252 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:12:33.391 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:12:33.411 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:12:33.411 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:12:33.411 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:12:33.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:12:33.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:12:33.651 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:12:33.881 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:12:33.881 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:12:33.919 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:12:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:12:33.984 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T19:12:33.986 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:12:34.026 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:12:35.101 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:14:35.162 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:14:35.200 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:14:36.237 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:14:36.254 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:14:36.254 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:14:36.254 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:14:36.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:14:36.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:14:36.512 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:14:36.722 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:14:36.723 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:14:36.762 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:14:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:14:36.827 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T19:14:36.828 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:14:36.864 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:14:37.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:16:37.938 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:16:37.976 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:16:39.059 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:16:39.087 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:16:39.087 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:16:39.087 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:16:39.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:16:39.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:16:39.328 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:16:39.560 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:16:39.560 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:16:39.596 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:16:39.630 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:16:39.661 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T19:16:39.662 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:16:39.700 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:16:40.740 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:18:40.790 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:18:40.830 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:18:41.860 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:18:41.878 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:18:41.879 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:18:41.879 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:18:41.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:18:42.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:18:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:18:42.366 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:18:42.367 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:18:42.406 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:18:42.436 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:18:42.475 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T19:18:42.475 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:18:42.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:18:43.621 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:20:43.668 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:20:43.705 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:20:44.802 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:20:44.819 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:20:44.819 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:20:44.819 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:20:44.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:20:45.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:20:45.084 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:20:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:20:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:20:45.355 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:20:45.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:20:45.422 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T19:20:45.423 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:20:45.462 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:20:46.561 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:22:46.610 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:22:46.648 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:22:47.734 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:22:47.751 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:22:47.751 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:22:47.751 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:22:47.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:22:47.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:22:47.996 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:22:51.046 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T19:22:51.103 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:22:52.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:24:52.215 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:24:52.253 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:24:53.312 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:24:53.330 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:24:53.330 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:24:53.330 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:24:53.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:24:53.542 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:24:53.571 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:24:53.600 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T19:24:54.698 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T19:24:55.759 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T19:24:56.882 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T19:24:57.981 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T19:24:59.122 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T19:25:00.218 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T19:25:01.275 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T19:25:02.373 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T19:25:03.432 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T19:25:03.432 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T19:25:03.469 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:25:04.605 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:27:04.649 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:27:04.686 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:27:06.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:27:06.558 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:27:06.558 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:27:06.558 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:27:06.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:27:06.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:27:06.816 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:27:38.179 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T19:27:38.219 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:27:39.272 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:29:39.321 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:29:39.359 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:29:40.473 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:29:40.490 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:29:40.490 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:29:40.490 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:29:40.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:29:40.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:29:40.780 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:29:40.945 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T19:29:40.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:29:40.980 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:29:40.996 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:29:40.996 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:29:40.998 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:29:41.038 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T19:29:41.050 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:29:41.119 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:29:41.274 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:29:41.296 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:29:41.296 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:29:41.297 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:29:41.335 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T19:29:41.350 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:29:41.420 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:29:41.421 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:29:41.437 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:29:41.437 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:29:41.488 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:29:42.618 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:31:42.670 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:31:43.907 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:31:45.784 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:31:45.801 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:31:45.801 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:31:45.801 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:31:45.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:31:46.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:31:46.040 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:32:01.347 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T19:32:01.385 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:32:02.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:34:02.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:34:03.061 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:34:04.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:34:04.223 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:34:04.223 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:34:04.224 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:34:04.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:34:04.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:34:04.481 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:34:04.625 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T19:34:04.663 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:34:04.663 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:34:04.698 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:34:04.698 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:34:04.701 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:34:04.732 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:34:04.800 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:34:04.800 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:34:04.838 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:34:04.859 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T19:34:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:34:04.937 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:34:04.937 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:34:04.954 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:34:04.954 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:34:04.994 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:34:06.043 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:36:06.094 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:36:06.140 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:36:07.238 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:36:07.254 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:36:07.254 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:36:07.255 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:36:07.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:36:07.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:36:07.496 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:36:07.651 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T19:36:07.681 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:36:07.681 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:36:07.699 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:36:07.699 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:36:07.701 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:36:07.750 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:36:07.818 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:36:07.819 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:36:07.863 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:36:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:36:07.897 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T19:36:07.962 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:36:07.962 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:36:07.979 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:36:07.979 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:36:08.020 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:36:09.096 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:38:09.146 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:38:09.182 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:38:10.273 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:38:10.292 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:38:10.292 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:38:10.292 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:38:10.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:38:10.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:38:10.539 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:38:10.696 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T19:38:10.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:38:10.731 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:38:10.749 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:38:10.749 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:38:10.751 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:38:10.800 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:38:10.870 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:38:10.871 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:38:10.910 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:38:10.943 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:38:11.011 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:38:11.011 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:38:11.092 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:38:12.159 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:40:12.208 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:40:12.245 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:40:13.329 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:40:13.349 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:40:13.349 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:40:13.349 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:40:13.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:40:13.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:40:13.607 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:40:13.755 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T19:40:13.787 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:40:13.788 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:40:13.804 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:40:13.804 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:40:13.807 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:40:13.859 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:40:13.926 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:40:13.926 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:40:13.964 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:40:13.989 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T19:40:13.995 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:40:14.026 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:40:14.064 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:40:15.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:42:15.242 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:42:15.283 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:42:16.421 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:42:16.437 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:42:16.437 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:42:16.437 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:42:16.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:42:16.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:42:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:42:16.839 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T19:42:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:42:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:42:16.891 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:42:16.891 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:42:16.892 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:42:16.946 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:42:17.018 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:42:17.019 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:42:17.061 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:42:17.088 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T19:42:17.088 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/3e5bbcd0-88ea-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T19:42:17.095 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:42:17.148 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:42:17.184 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:42:18.223 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:44:18.272 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:44:18.333 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:44:20.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:44:20.215 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:44:20.215 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:44:20.215 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:44:20.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:44:20.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:44:20.456 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:44:20.636 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T19:44:20.691 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:44:20.691 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:44:20.716 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:44:20.717 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:44:20.719 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:44:20.787 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:44:20.861 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:44:20.861 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:44:20.910 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:44:20.938 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T19:44:20.938 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/3e5bbcd0-88ea-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T19:44:20.940 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:44:20.942 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:44:20.982 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:44:22.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:46:22.752 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:46:22.790 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:46:23.923 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:46:23.941 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:46:23.941 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:46:23.941 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:46:23.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:46:24.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:46:24.200 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:46:24.336 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T19:46:24.365 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:46:24.366 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:46:24.381 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:46:24.382 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:46:24.384 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:46:24.435 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:46:24.503 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:46:24.503 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:46:24.545 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:46:24.571 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:46:24.587 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T19:46:24.588 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/3e5bbcd0-88ea-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T19:46:24.590 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T19:46:24.631 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:46:25.750 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:48:25.805 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:48:25.848 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:48:26.894 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:48:26.910 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:48:26.911 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:48:26.911 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:48:26.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:48:27.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:48:27.153 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:49:27.126 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T19:49:27.170 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:49:28.220 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:51:28.267 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:51:28.313 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:51:29.504 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:51:29.523 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:51:29.523 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:51:29.524 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:51:29.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:51:29.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:51:29.778 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:51:48.162 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T19:51:48.220 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:51:50.083 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:53:50.132 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:53:50.175 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:53:52.346 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:53:52.364 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:53:52.364 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:53:52.364 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:53:52.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:53:52.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:53:52.610 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:53:55.106 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T19:53:55.151 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:53:57.469 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:55:57.523 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:55:57.563 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:55:59.207 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:55:59.225 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:55:59.225 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:55:59.225 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:55:59.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:55:59.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:55:59.528 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:55:59.833 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T19:55:59.866 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:55:59.866 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:55:59.892 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:55:59.893 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:55:59.895 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:55:59.937 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:55:59.953 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T19:56:00.008 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:56:00.008 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:56:00.025 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:56:00.025 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:56:00.027 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:56:00.077 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:56:00.084 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T19:56:00.148 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:56:00.149 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:56:00.172 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:56:00.172 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:56:00.236 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:56:02.397 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:58:02.451 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T19:58:02.491 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:58:04.043 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T19:58:04.060 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T19:58:04.061 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T19:58:04.061 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:58:04.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:58:04.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T19:58:04.321 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T19:58:04.638 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T19:58:04.671 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:58:04.672 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:58:04.695 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T19:58:04.695 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:58:04.697 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:58:04.742 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T19:58:04.755 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T19:58:04.811 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:58:04.812 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:58:04.828 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T19:58:04.828 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:58:04.830 INFO:teuthology.misc:Compressing logs... 2021-03-19T19:58:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T19:58:04.887 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T19:58:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:58:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:58:04.965 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T19:58:04.965 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T19:58:05.009 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T19:58:06.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:00:06.226 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:00:06.267 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:00:07.507 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:00:07.526 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:00:07.527 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:00:07.527 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:00:07.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:00:07.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:00:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:00:08.097 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T20:00:08.129 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:00:08.129 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:00:08.159 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:00:08.159 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:00:08.160 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:00:08.199 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:00:08.271 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:00:08.271 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:00:08.306 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:00:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:00:08.348 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T20:00:08.410 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:00:08.411 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:00:08.429 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:00:08.429 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:00:08.492 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:00:09.683 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:02:09.745 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:02:09.869 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:02:10.969 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:02:10.991 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:02:10.991 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:02:10.991 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:02:10.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:02:11.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:02:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:02:11.519 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T20:02:11.550 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:02:11.550 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:02:11.576 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:02:11.576 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:02:11.578 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:02:11.619 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:02:11.688 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:02:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:02:11.733 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:02:11.762 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:02:11.765 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T20:02:11.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:02:11.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:02:11.851 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:02:11.851 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:02:11.911 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:02:13.072 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:04:13.120 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:04:13.159 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:04:14.222 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:04:14.238 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:04:14.238 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:04:14.238 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:04:14.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:04:14.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:04:14.501 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:04:14.815 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:04:14.815 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:04:14.844 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:04:14.884 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:04:14.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:04:14.953 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:04:14.998 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:04:15.024 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:04:15.092 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:04:15.092 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:04:15.148 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:04:16.205 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:06:16.258 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:06:16.318 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:06:17.357 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:06:17.375 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:06:17.375 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:06:17.376 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:06:17.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:06:17.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:06:17.625 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:06:17.963 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:06:17.963 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:06:17.992 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:06:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:06:18.102 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:06:18.103 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:06:18.139 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:06:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:06:18.289 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:06:18.289 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:06:18.430 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:06:19.470 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:08:19.538 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:08:19.577 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:08:20.629 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:08:20.646 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:08:20.647 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:08:20.647 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:08:20.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:08:20.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:08:20.890 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:08:21.233 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:08:21.233 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:08:21.261 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:08:21.303 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:08:21.372 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:08:21.372 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:08:21.413 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:08:21.441 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:08:21.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.11.log.gz already exists; not overwritten 2021-03-19T20:08:21.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.9.log.gz already exists; not overwritten 2021-03-19T20:08:21.453 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-19T20:08:21.453 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.8.log.gz already exists; not overwritten 2021-03-19T20:08:21.453 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-19T20:08:21.453 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.10.log.gz already exists; not overwritten 2021-03-19T20:08:21.454 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T20:08:21.492 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:08:22.574 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:10:22.627 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:10:22.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:10:23.732 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:10:23.749 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:10:23.750 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:10:23.750 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:10:23.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:10:23.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:10:24.007 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:10:42.503 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T20:10:42.542 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:10:43.591 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:12:43.636 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:12:43.676 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:12:44.759 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:12:44.777 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:12:44.777 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:12:44.777 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:12:44.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:12:44.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:12:45.020 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:12:48.066 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T20:12:48.103 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:12:49.204 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:14:49.260 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:14:49.300 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:14:50.384 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:14:50.402 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:14:50.402 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:14:50.402 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:14:50.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:14:50.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:14:50.644 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:14:51.196 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T20:14:51.229 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:14:51.230 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:14:51.256 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:14:51.256 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:14:51.258 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:14:51.301 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:14:51.320 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T20:14:51.372 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:14:51.372 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:14:51.395 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:14:51.395 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:14:51.397 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:14:51.460 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:14:51.477 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T20:14:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:14:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:14:51.577 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:14:51.577 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:14:51.615 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:14:52.653 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:16:52.700 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:16:52.738 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:16:53.823 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:16:53.840 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:16:53.840 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:16:53.840 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:16:53.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:16:54.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:16:54.104 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:17:54.062 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T20:17:54.102 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:17:55.167 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:19:55.216 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:19:55.315 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:19:57.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:19:57.283 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:19:57.283 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:19:57.284 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:19:57.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:19:57.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:19:57.528 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:19:57.890 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T20:19:57.927 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:19:57.927 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:19:57.951 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:19:57.951 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:19:57.953 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:19:57.999 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:19:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:19:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:19:58.110 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:19:58.137 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:19:58.167 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T20:19:58.207 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:19:58.208 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:19:58.225 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:19:58.226 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:19:58.266 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:20:00.005 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:22:00.057 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:22:00.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:22:01.282 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:22:01.302 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:22:01.302 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:22:01.302 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:22:01.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:22:01.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:22:01.567 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:22:01.828 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T20:22:01.862 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:22:01.863 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:22:01.887 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:22:01.887 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:22:01.889 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:22:01.933 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:22:02.004 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:22:02.004 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:22:02.034 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:22:02.075 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:22:02.080 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T20:22:02.146 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:22:02.146 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:22:02.164 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:22:02.164 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:22:02.205 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:22:03.308 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:24:03.359 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:24:03.397 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:24:04.480 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:24:04.497 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:24:04.497 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:24:04.497 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:24:04.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:24:04.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:24:04.742 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:24:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:24:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:24:05.103 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:24:05.142 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:24:05.214 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:24:05.214 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:24:05.250 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:24:05.283 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:24:05.353 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:24:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:24:05.415 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:24:06.477 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:26:06.527 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:26:06.567 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:26:07.621 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:26:07.637 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:26:07.638 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:26:07.638 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:26:07.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:26:07.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:26:07.885 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:26:08.232 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:26:08.232 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:26:08.267 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:26:08.303 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:26:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:26:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:26:08.411 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:26:08.442 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:26:08.961 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping 2021-03-19T20:26:09.202 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2021-03-19T20:26:09.651 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2021-03-19T20:26:10.600 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping 2021-03-19T20:26:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:26:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:26:11.255 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:26:12.411 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:28:12.458 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:28:12.501 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:28:13.605 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:28:13.623 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:28:13.624 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:28:13.624 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:28:13.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:28:13.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:28:13.887 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:28:14.213 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:28:14.213 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:28:14.245 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:28:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:28:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:28:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:28:14.391 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:28:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:28:14.443 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-19T20:28:14.444 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-19T20:28:14.445 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T20:28:14.482 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:28:15.502 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:30:15.548 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:30:15.594 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:30:16.717 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:30:16.734 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:30:16.735 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:30:16.735 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:30:16.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:30:16.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:30:16.986 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:30:17.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:30:17.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:30:17.357 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:30:17.398 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:30:17.466 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:30:17.467 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:30:17.501 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:30:17.536 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:30:17.560 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-19T20:30:17.560 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-19T20:30:17.563 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T20:30:17.608 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:30:18.710 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:32:18.759 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:32:18.820 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:32:20.860 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:32:20.879 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:32:20.879 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:32:20.879 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:32:20.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:32:21.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:32:21.125 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:32:21.459 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:32:21.460 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:32:21.490 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:32:21.529 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:32:21.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:32:21.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:32:21.658 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:32:21.686 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:32:21.717 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-19T20:32:21.717 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-19T20:32:21.717 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T20:32:21.785 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:32:23.913 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:34:23.971 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:34:24.013 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:34:25.125 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:34:25.148 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:34:25.148 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:34:25.149 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:34:25.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:34:25.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:34:25.409 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:34:25.723 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:34:25.723 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:34:25.754 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:34:25.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:34:25.861 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:34:25.861 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:34:25.904 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:34:25.931 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:34:25.943 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-19T20:34:25.943 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-19T20:34:25.946 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T20:34:25.983 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:34:27.118 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:36:27.178 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:36:27.219 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:36:28.549 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:36:28.568 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:36:28.568 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:36:28.569 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:36:28.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:36:28.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:36:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:36:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:36:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:36:29.191 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:36:29.230 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:36:29.299 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:36:29.300 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:36:29.338 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:36:29.368 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:36:29.380 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-19T20:36:29.380 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-19T20:36:29.381 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T20:36:29.441 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:36:30.565 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:38:30.613 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:38:30.699 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:38:32.619 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:38:32.641 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:38:32.641 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:38:32.641 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:38:32.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:38:32.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:38:32.894 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:38:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:38:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:38:33.265 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:38:33.303 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:38:33.373 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:38:33.374 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:38:33.416 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:38:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:38:33.473 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-19T20:38:33.473 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-19T20:38:33.475 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T20:38:33.521 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:38:34.621 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:40:34.668 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:40:34.709 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:40:35.856 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:40:35.873 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:40:35.873 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:40:35.873 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:40:35.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:40:36.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:40:36.131 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:41:36.089 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T20:41:36.130 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:41:37.997 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:43:38.052 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:43:38.098 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:43:39.232 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:43:39.249 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:43:39.249 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:43:39.249 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:43:39.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:43:39.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:43:39.492 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:43:39.565 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T20:43:40.668 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T20:43:41.731 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T20:43:42.830 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T20:43:43.928 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T20:43:45.026 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T20:43:46.124 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T20:43:47.221 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T20:43:48.320 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T20:43:49.417 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-19T20:43:49.417 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T20:43:49.457 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:43:50.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:45:50.687 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:45:50.724 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:45:52.477 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:45:52.494 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:45:52.494 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:45:52.494 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:45:52.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:45:52.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:45:52.752 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:45:55.618 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T20:45:55.658 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:45:57.402 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:47:57.472 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:47:57.511 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:47:58.573 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:47:58.590 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:47:58.590 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:47:58.590 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:47:58.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:47:58.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:47:58.837 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:47:59.141 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T20:47:59.175 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:47:59.175 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:47:59.200 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:47:59.201 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:47:59.204 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:47:59.246 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:47:59.265 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-19T20:47:59.316 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:47:59.316 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:47:59.332 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:47:59.332 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:47:59.334 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:47:59.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:47:59.389 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T20:47:59.455 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:47:59.456 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:47:59.472 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:47:59.472 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:47:59.517 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:48:00.571 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:50:00.622 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:50:00.661 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:50:01.745 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:50:01.763 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:50:01.763 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:50:01.763 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:50:01.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:50:01.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:50:02.006 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:50:20.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T20:50:20.526 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:50:21.637 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:52:21.685 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:52:21.826 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:52:23.690 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:52:23.707 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:52:23.707 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:52:23.707 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:52:23.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:52:23.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:52:23.964 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:52:24.322 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T20:52:24.357 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:52:24.357 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:52:24.373 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:52:24.373 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:52:24.375 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:52:24.428 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:52:24.498 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:52:24.499 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:52:24.536 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:52:24.567 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:52:24.576 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T20:52:24.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:52:24.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:52:24.656 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:52:24.656 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:52:24.694 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:52:25.828 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:54:25.877 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:54:25.917 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:54:27.015 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:54:27.033 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:54:27.033 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:54:27.033 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:54:27.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:54:27.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:54:27.274 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:54:27.544 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-19T20:54:27.577 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:54:27.577 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:54:27.603 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:54:27.603 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:54:27.605 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:54:27.648 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:54:27.717 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:54:27.717 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:54:27.756 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:54:27.786 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:54:27.810 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-19T20:54:27.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:54:27.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:54:27.872 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:54:27.872 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:54:27.910 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:54:28.964 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:56:29.012 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:56:29.049 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:56:30.109 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:56:30.126 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:56:30.126 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:56:30.126 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:56:30.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:56:30.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:56:30.376 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:56:30.710 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:56:30.710 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:56:30.743 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:56:30.781 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:56:30.852 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:56:30.852 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:56:30.892 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:56:30.922 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:56:30.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:56:30.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:56:31.052 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:56:32.174 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:58:32.222 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T20:58:32.274 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:58:34.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T20:58:34.256 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T20:58:34.256 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T20:58:34.256 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:58:34.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:58:34.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T20:58:34.519 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T20:58:34.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T20:58:34.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:58:34.850 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:58:34.893 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T20:58:34.962 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T20:58:34.962 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:58:34.993 INFO:teuthology.misc:Compressing logs... 2021-03-19T20:58:35.030 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T20:58:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T20:58:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T20:58:35.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T20:58:36.768 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:00:36.816 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:00:36.862 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:00:38.061 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:00:38.097 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:00:38.097 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:00:38.097 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:00:38.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:00:38.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:00:38.355 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:00:38.688 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:00:38.689 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:00:38.718 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:00:38.759 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:00:38.830 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T21:00:38.831 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:00:38.878 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:00:38.906 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:00:38.915 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T21:00:38.947 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:00:38.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:00:40.163 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:02:40.222 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:02:40.259 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:02:41.387 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:02:41.404 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:02:41.404 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:02:41.404 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:02:41.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:02:41.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:02:41.655 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:02:42.108 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:02:42.108 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:02:42.134 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:02:42.179 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:02:42.248 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T21:02:42.248 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:02:42.289 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:02:42.317 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:02:42.344 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T21:02:42.350 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/74650ce0-88f5-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T21:02:42.351 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:02:42.396 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:02:43.488 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:04:43.539 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:04:43.584 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:04:44.668 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:04:44.688 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:04:44.688 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:04:44.688 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:04:44.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:04:44.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:04:44.951 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:04:45.273 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:04:45.273 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:04:45.304 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:04:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:04:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T21:04:45.414 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:04:45.453 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:04:45.482 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:04:45.496 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T21:04:45.543 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/74650ce0-88f5-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-19T21:04:45.543 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/74650ce0-88f5-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T21:04:45.565 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:04:45.604 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:04:46.706 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:06:46.757 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:06:46.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:06:47.830 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:06:47.849 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:06:47.850 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:06:47.850 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:06:47.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:06:48.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:06:48.094 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:06:48.434 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:06:48.434 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:06:48.465 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:06:48.505 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:06:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T21:06:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:06:48.619 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:06:48.647 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:06:48.677 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-19T21:06:48.677 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/74650ce0-88f5-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-19T21:06:48.677 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/74650ce0-88f5-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-19T21:06:48.679 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:06:48.716 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:06:49.819 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:08:49.868 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:08:49.906 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:08:50.999 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:08:51.018 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:08:51.018 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:08:51.018 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:08:51.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:08:51.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:08:51.265 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:09:09.699 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T21:09:09.738 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:09:10.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:11:10.877 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:11:10.921 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:11:12.017 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:11:12.035 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:11:12.035 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:11:12.035 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:11:12.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:11:12.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:11:12.341 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:11:30.759 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T21:11:30.801 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:11:31.900 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:13:31.947 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:13:31.987 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:13:33.114 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:13:33.131 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:13:33.132 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:13:33.132 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:13:33.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:13:33.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:13:33.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:13:36.422 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T21:13:36.523 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:13:37.615 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:15:37.665 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:15:37.702 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:15:38.727 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:15:38.744 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:15:38.744 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:15:38.744 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:15:38.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:15:38.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:15:38.986 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:15:39.184 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:15:39.216 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:15:39.322 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:15:39.376 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:15:39.377 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:15:39.378 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:15:39.407 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:15:39.478 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T21:15:39.478 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:15:39.534 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:15:39.568 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:15:39.626 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:15:39.655 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T21:15:39.655 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:15:39.702 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T21:15:39.702 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:15:39.740 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:15:40.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:17:40.866 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:17:40.911 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:17:41.986 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:17:42.006 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:17:42.006 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:17:42.006 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:17:42.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:17:42.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:17:42.264 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:17:42.445 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:17:42.477 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:17:42.477 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:17:42.510 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:17:42.510 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:17:42.511 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:17:42.546 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:17:42.617 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T21:17:42.618 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:17:42.658 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:17:42.685 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:17:42.709 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:17:42.754 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T21:17:42.754 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:17:42.779 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T21:17:42.780 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:17:42.817 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:17:43.861 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:19:43.909 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:19:43.955 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:19:45.070 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:19:45.088 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:19:45.088 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:19:45.088 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:19:45.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:19:45.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:19:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:19:45.537 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:19:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:19:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:19:45.604 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:19:45.604 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:19:45.605 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:19:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:19:45.670 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:19:45.671 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:19:45.711 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:19:46.801 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:21:46.849 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:21:46.888 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:21:48.954 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:21:48.983 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:21:48.983 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:21:48.983 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:21:48.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:21:49.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:21:49.240 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:21:49.479 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:21:49.480 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:21:49.521 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:21:49.550 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:21:49.586 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:21:49.588 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:21:49.637 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:21:50.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:23:51.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:23:51.064 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:23:52.585 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:23:52.604 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:23:52.605 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:23:52.605 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:23:52.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:23:52.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:23:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:23:53.157 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:23:53.157 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:23:53.203 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:23:53.235 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:23:53.277 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:23:53.279 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:23:53.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:23:55.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:25:55.122 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:25:55.163 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:25:56.270 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:25:56.288 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:25:56.288 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:25:56.288 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:25:56.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:25:56.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:25:56.532 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:25:56.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:25:56.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:25:56.932 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:25:56.960 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:25:56.983 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:25:56.985 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:25:57.027 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:25:58.201 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:27:58.260 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:27:58.299 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:27:59.427 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:27:59.444 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:27:59.444 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:27:59.445 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:27:59.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:27:59.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:27:59.688 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:28:18.115 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T21:28:18.153 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:28:19.224 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:30:19.277 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:30:19.319 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:30:20.348 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:30:20.367 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:30:20.367 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:30:20.367 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:30:20.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:30:20.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:30:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:30:38.918 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T21:30:38.975 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:30:40.135 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:32:40.186 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:32:40.224 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:32:41.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:32:41.301 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:32:41.301 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:32:41.301 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:32:41.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:32:41.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:32:41.545 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:32:41.602 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T21:32:41.641 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:32:42.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:34:42.754 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:34:42.799 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:34:43.905 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:34:43.922 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:34:43.922 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:34:43.922 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:34:43.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:34:44.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:34:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:34:44.371 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:34:44.405 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:34:44.405 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:34:44.449 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:34:44.450 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:34:44.451 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:34:44.480 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:34:44.516 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:34:44.552 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T21:34:44.552 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:34:44.583 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T21:34:44.583 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:34:44.584 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:34:44.621 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:34:44.648 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:34:44.690 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T21:34:44.690 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:34:44.715 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T21:34:44.715 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:34:44.754 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:34:45.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:36:45.898 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:36:45.937 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:36:48.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:36:48.043 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:36:48.043 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:36:48.043 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:36:48.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:36:48.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:36:48.268 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T21:36:48.340 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:36:50.101 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:38:50.148 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:38:50.189 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:38:52.211 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:38:52.231 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:38:52.232 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:38:52.232 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:38:52.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:38:52.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:38:52.477 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:38:52.686 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:38:52.715 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:38:52.715 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:38:52.761 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:38:52.761 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:38:52.763 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:38:52.790 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:38:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T21:38:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:38:52.910 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:38:52.936 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:38:52.964 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:38:53.003 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T21:38:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:38:53.030 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T21:38:53.030 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:38:53.072 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:38:54.322 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:40:54.373 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:40:54.409 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:40:55.434 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:40:55.451 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:40:55.451 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:40:55.451 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:40:55.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:40:55.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:40:55.784 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:40:55.985 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:40:56.015 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:40:56.016 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:40:56.051 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:40:56.051 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:40:56.052 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:40:56.086 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:40:56.155 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-19T21:40:56.155 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:40:56.197 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:40:56.225 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:40:56.246 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-19T21:40:56.293 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T21:40:56.294 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:40:56.318 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-19T21:40:56.319 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:40:56.358 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:40:57.369 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:42:57.415 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:42:57.459 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:42:58.560 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:42:58.578 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:42:58.578 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:42:58.578 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:42:58.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:42:58.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:42:58.837 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:42:59.022 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:42:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:42:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:42:59.090 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:42:59.091 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:42:59.092 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:42:59.123 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:42:59.157 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:42:59.158 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:42:59.202 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:43:00.336 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:45:00.397 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:45:00.444 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:45:02.423 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:45:02.605 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:45:02.606 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:45:02.606 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:45:02.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:45:02.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:45:02.859 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:45:03.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:45:03.092 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:45:03.132 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:45:03.161 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:45:03.197 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:45:03.199 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:45:03.249 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:45:05.032 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:47:05.079 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:47:05.878 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:47:06.952 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:47:06.968 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:47:06.968 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:47:06.968 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:47:06.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:47:07.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:47:07.396 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:47:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:47:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:47:07.686 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:47:07.726 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:47:07.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:47:07.754 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:47:07.793 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:47:09.100 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:49:09.161 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:49:09.201 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:49:10.270 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:49:10.287 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:49:10.287 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:49:10.287 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:49:10.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:49:10.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:49:10.550 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:49:10.820 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:49:10.820 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:49:10.860 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:49:10.889 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:49:10.926 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:49:10.928 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:49:11.030 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:49:12.133 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:51:12.181 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:51:12.217 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:51:14.200 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:51:14.217 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:51:14.217 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:51:14.218 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:51:14.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:51:14.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:51:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:51:14.701 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:51:14.701 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:51:14.741 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:51:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:51:14.806 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:51:14.807 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:51:14.846 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:51:16.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:53:16.874 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:53:16.915 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:53:17.986 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:53:18.006 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:53:18.006 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:53:18.006 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:53:18.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:53:18.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:53:18.252 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:53:18.481 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:53:18.481 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:53:18.519 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:53:18.553 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:53:18.585 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:53:18.587 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:53:18.624 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:53:19.658 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:55:19.704 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:55:19.741 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:55:20.851 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:55:20.867 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:55:20.867 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:55:20.867 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:55:20.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:55:21.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:55:21.128 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:55:21.346 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:55:21.346 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:55:21.386 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:55:21.415 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:55:21.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:55:21.453 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:55:21.491 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:55:22.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:57:22.650 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:57:22.689 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:57:23.751 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:57:23.768 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:57:23.768 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:57:23.769 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:57:23.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:57:23.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:57:24.016 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:57:24.261 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:57:24.261 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:57:24.290 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:57:24.333 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:57:24.356 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:57:24.357 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:57:24.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:57:25.490 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:59:25.550 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:59:25.591 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:59:26.691 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T21:59:26.711 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T21:59:26.711 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T21:59:26.711 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:59:26.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:59:26.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:59:26.953 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T21:59:27.188 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T21:59:27.188 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T21:59:27.227 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:59:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T21:59:27.294 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T21:59:27.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T21:59:27.352 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T21:59:28.506 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:01:28.560 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:01:28.602 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:01:29.684 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:01:29.701 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:01:29.701 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:01:29.701 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:01:29.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:01:29.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:01:29.958 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:01:30.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:01:30.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:01:30.221 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:01:30.251 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:01:30.286 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:01:30.287 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:01:30.325 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:01:31.383 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:03:31.432 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:03:31.470 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:03:32.538 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:03:32.557 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:03:32.557 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:03:32.557 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:03:32.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:03:32.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:03:32.805 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:03:33.058 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:03:33.058 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:03:33.088 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:03:33.128 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:03:33.154 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:03:33.160 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:03:33.200 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:03:34.303 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:05:34.350 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:05:34.390 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:05:35.447 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:05:35.466 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:05:35.467 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:05:35.467 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:05:35.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:05:35.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:05:35.712 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:05:35.945 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:05:35.946 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:05:35.983 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:05:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:05:36.049 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:05:36.050 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:05:36.089 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:05:37.158 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:07:37.207 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:07:37.245 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:07:38.278 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:07:38.295 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:07:38.295 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:07:38.296 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:07:38.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:07:38.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:07:38.555 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:07:38.767 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:07:38.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:07:38.808 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:07:38.836 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:07:38.874 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:07:38.876 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:07:38.914 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:07:39.981 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:09:40.034 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:09:40.082 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:09:41.097 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:09:41.113 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:09:41.113 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:09:41.113 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:09:41.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:09:41.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:09:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:09:41.597 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:09:41.597 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:09:41.626 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:09:41.666 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:09:41.691 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:09:41.693 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:09:41.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:09:42.750 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:11:42.798 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:11:42.855 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:11:43.872 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:11:43.899 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:11:43.899 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:11:43.900 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:11:43.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:11:44.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:11:44.149 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:11:44.384 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:11:44.384 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:11:44.422 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:11:44.456 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:11:44.488 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:11:44.489 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:11:44.527 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:11:45.550 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:13:45.598 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:13:45.634 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:13:46.694 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:13:46.711 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:13:46.712 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:13:46.712 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:13:46.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:13:46.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:13:46.971 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:13:47.184 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:13:47.184 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:13:47.225 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:13:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:13:47.290 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:13:47.292 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:13:47.329 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:13:48.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:15:48.415 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:15:48.454 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:15:49.486 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:15:49.503 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:15:49.503 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:15:49.504 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:15:49.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:15:49.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:15:49.745 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:15:49.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:15:49.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:15:50.021 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:15:50.063 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:15:50.087 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:15:50.088 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:15:50.126 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:15:51.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:17:51.240 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:17:51.283 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:17:52.351 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:17:52.380 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:17:52.380 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:17:52.380 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:17:52.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:17:52.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:17:52.624 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:17:52.852 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:17:52.853 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:17:52.890 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:17:52.924 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:17:52.954 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:17:52.956 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:17:52.993 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:17:54.031 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:19:54.079 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:19:54.121 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:19:55.190 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:19:55.208 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:19:55.209 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:19:55.209 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:19:55.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:19:55.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:19:55.468 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:19:55.726 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:19:55.726 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:19:55.760 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:19:55.799 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:19:55.831 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:19:55.833 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:19:55.870 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:19:56.947 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:21:57.001 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:21:57.036 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:21:58.081 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:21:58.098 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:21:58.098 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:21:58.098 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:21:58.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:21:58.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:21:58.345 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:21:58.597 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:21:58.597 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:21:58.626 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:21:58.667 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:21:58.692 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:21:58.694 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:21:58.731 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:21:59.783 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:23:59.831 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:23:59.869 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:24:00.910 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:24:00.929 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:24:00.930 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:24:00.930 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:24:00.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:24:01.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:24:01.171 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:24:01.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:24:01.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:24:01.447 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:24:01.477 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:24:01.512 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:24:01.514 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:24:01.589 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:24:02.614 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:26:02.660 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:26:02.700 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:26:03.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:26:03.710 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:26:03.711 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:26:03.711 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:26:03.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:26:03.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:26:03.974 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:26:04.185 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:26:04.186 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:26:04.226 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:26:04.254 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:26:04.291 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:26:04.293 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:26:04.331 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:26:05.384 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:28:05.433 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:28:05.471 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:28:06.572 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:28:06.588 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:28:06.589 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:28:06.589 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:28:06.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:28:06.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:28:06.841 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:28:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:28:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:28:07.127 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:28:07.155 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:28:07.192 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:28:07.194 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:28:07.231 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:28:08.268 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:30:08.312 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:30:08.352 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:30:10.169 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:30:10.187 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:30:10.187 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:30:10.188 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:30:10.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:30:10.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:30:10.430 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:30:10.667 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:30:10.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:30:10.706 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:30:10.736 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:30:10.774 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:30:10.775 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:30:10.848 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:30:11.956 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:32:12.004 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:32:12.043 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:32:13.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:32:13.812 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:32:13.813 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:32:13.813 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:32:13.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:32:14.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:32:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:32:14.300 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:32:14.300 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:32:14.344 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:32:14.372 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:32:14.410 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:32:14.411 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:32:14.468 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:32:15.523 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:34:15.573 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:34:15.611 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:34:16.665 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:34:16.682 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:34:16.682 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:34:16.683 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:34:16.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:34:16.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:34:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:34:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:34:17.180 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:34:17.209 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:34:17.249 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:34:17.276 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:34:17.277 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:34:17.311 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:34:18.325 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:36:18.373 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:36:18.418 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:36:20.215 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:36:20.232 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:36:20.233 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:36:20.233 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:36:20.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:36:20.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:36:20.477 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:36:20.714 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:36:20.714 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:36:20.753 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:36:20.783 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:36:20.819 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:36:20.821 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:36:20.860 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:36:22.303 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:38:22.366 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:38:22.432 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:38:24.276 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:38:24.296 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:38:24.296 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:38:24.296 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:38:24.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:38:24.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:38:24.559 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:38:24.790 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:38:24.790 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:38:24.830 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:38:24.860 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:38:24.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:38:24.897 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:38:24.941 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:38:26.487 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:40:26.543 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:40:26.584 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:40:27.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:40:27.710 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:40:27.710 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:40:27.711 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:40:27.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:40:27.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:40:27.958 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:40:28.202 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:40:28.202 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:40:28.240 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:40:28.271 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:40:28.306 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:40:28.307 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:40:28.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:40:29.416 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:42:29.476 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:42:29.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:42:30.553 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:42:30.570 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:42:30.571 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:42:30.571 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:42:30.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:42:30.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:42:30.812 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:42:31.039 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:42:31.040 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:42:31.078 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:42:31.111 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:42:31.142 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:42:31.143 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:42:31.180 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:42:32.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:44:32.846 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:44:32.885 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:44:34.004 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:44:34.021 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:44:34.021 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:44:34.022 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:44:34.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:44:34.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:44:34.283 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:44:34.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:44:34.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:44:34.535 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:44:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:44:34.601 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:44:34.602 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:44:34.646 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:44:35.754 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:46:35.804 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:46:35.844 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:46:36.951 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:46:36.968 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:46:36.968 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:46:36.968 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:46:36.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:46:37.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:46:37.214 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:46:37.473 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:46:37.473 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:46:37.503 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:46:37.545 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:46:37.569 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:46:37.570 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:46:37.615 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:46:38.690 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:48:38.739 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:48:38.778 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:48:40.498 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:48:40.515 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:48:40.516 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:48:40.516 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:48:40.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:48:40.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:48:40.763 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:48:41.003 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:48:41.003 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:48:41.041 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:48:41.072 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:48:41.107 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:48:41.108 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:48:41.147 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:48:42.257 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:50:42.316 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:50:42.354 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:50:43.534 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:50:43.551 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:50:43.551 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:50:43.551 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:50:43.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:50:43.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:50:43.825 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:50:44.043 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:50:44.044 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:50:44.083 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:50:44.112 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:50:44.149 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:50:44.150 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:50:44.190 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:50:45.195 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:52:45.415 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:52:45.456 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:52:46.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:52:46.558 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:52:46.558 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:52:46.558 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:52:46.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:52:46.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:52:46.803 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:52:47.054 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:52:47.055 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:52:47.083 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:52:47.125 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:52:47.149 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:52:47.151 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:52:47.189 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:52:48.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:54:48.333 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:54:48.372 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:54:49.401 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:54:49.419 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:54:49.419 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:54:49.419 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:54:49.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:54:49.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:54:49.670 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:54:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:54:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:54:49.935 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:54:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:54:50.000 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:54:50.002 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:54:50.048 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:54:51.089 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:56:51.137 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:56:51.175 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:56:52.217 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:56:52.235 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:56:52.235 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:56:52.235 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:56:52.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:56:52.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:56:52.504 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:56:52.714 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:56:52.714 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:56:52.755 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:56:52.782 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:56:52.819 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:56:52.821 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:56:52.858 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:56:53.873 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:58:53.923 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T22:58:54.014 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:58:54.993 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T22:58:55.008 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T22:58:55.008 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T22:58:55.008 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:58:55.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:58:55.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T22:58:55.248 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T22:58:55.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T22:58:55.492 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T22:58:55.528 INFO:teuthology.misc:Compressing logs... 2021-03-19T22:58:55.562 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T22:58:55.595 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T22:58:55.596 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T22:58:55.634 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T22:58:56.647 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:00:56.693 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:00:56.736 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:00:57.828 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:00:57.846 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:00:57.846 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:00:57.846 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:00:57.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:00:58.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:00:58.086 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:00:58.314 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:00:58.315 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:00:58.354 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:00:58.384 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:00:58.419 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:00:58.421 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:00:58.458 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:00:59.460 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:02:59.511 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:02:59.552 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:03:00.603 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:03:00.657 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:03:00.657 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:03:00.657 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:03:00.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:03:00.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:03:00.922 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:03:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:03:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:03:01.185 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:03:01.213 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:03:01.252 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:03:01.253 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:03:01.291 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:03:02.306 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:05:02.355 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:05:02.401 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:05:03.453 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:05:03.471 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:05:03.472 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:05:03.472 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:05:03.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:05:03.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:05:03.718 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:05:03.973 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:05:03.973 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:05:04.002 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:05:04.044 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:05:04.068 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:05:04.069 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:05:04.110 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:05:05.152 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:07:05.208 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:07:05.247 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:07:06.344 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:07:06.361 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:07:06.361 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:07:06.361 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:07:06.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:07:06.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:07:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:07:06.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:07:06.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:07:06.878 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:07:06.909 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:07:06.943 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:07:06.945 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:07:06.987 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:07:08.036 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:09:08.090 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:09:08.128 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:09:09.115 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:09:09.132 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:09:09.132 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:09:09.132 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:09:09.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:09:09.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:09:09.392 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:09:09.611 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:09:09.611 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:09:09.650 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:09:09.680 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:09:09.715 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:09:09.716 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:09:09.750 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:09:10.713 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:11:10.769 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:11:10.804 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:11:11.809 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:11:11.826 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:11:11.826 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:11:11.826 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:11:11.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:11:12.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:11:12.069 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:11:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:11:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:11:12.353 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:11:12.382 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:11:12.418 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:11:12.420 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:11:12.455 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:11:13.475 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:13:13.524 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:13:13.560 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:13:14.547 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:13:14.562 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:13:14.562 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:13:14.562 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:13:14.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:13:14.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:13:14.804 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:13:15.041 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:13:15.041 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:13:15.080 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:13:15.111 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:13:15.145 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:13:15.147 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:13:15.182 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:13:16.152 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:15:16.201 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:15:16.239 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:15:17.286 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:15:17.301 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:15:17.302 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:15:17.302 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:15:17.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:15:17.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:15:17.564 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:15:17.782 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:15:17.783 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:15:17.823 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:15:17.851 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:15:17.889 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:15:17.891 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:15:17.925 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:15:18.878 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:17:18.924 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:17:18.960 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:17:20.029 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:17:20.047 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:17:20.047 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:17:20.047 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:17:20.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:17:20.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:17:20.286 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:17:20.538 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:17:20.538 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:17:20.566 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:17:20.607 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:17:20.632 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:17:20.633 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:17:20.669 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:17:21.648 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:19:21.693 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:19:21.729 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:19:22.740 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:19:22.759 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:19:22.759 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:19:22.759 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:19:22.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:19:22.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:19:22.999 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:19:23.235 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:19:23.236 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:19:23.275 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:19:23.304 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:19:23.341 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:19:23.343 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:19:23.380 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:19:24.366 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:21:24.414 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:21:24.451 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:21:25.439 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:21:25.456 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:21:25.456 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:21:25.456 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:21:25.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:21:25.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:21:25.711 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:21:26.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:21:26.013 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:21:26.052 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:21:26.081 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:21:26.118 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:21:26.120 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:21:26.155 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:21:27.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:23:27.229 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:23:27.267 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:23:28.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:23:28.278 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:23:28.278 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:23:28.278 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:23:28.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:23:28.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:23:28.519 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:23:28.772 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:23:28.773 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:23:28.802 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:23:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:23:28.868 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:23:28.870 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:23:28.904 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:23:29.928 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:25:29.980 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:25:30.020 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:25:31.083 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:25:31.100 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:25:31.100 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:25:31.100 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:25:31.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:25:31.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:25:31.345 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:25:31.588 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:25:31.588 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:25:31.626 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:25:31.658 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:25:31.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:25:31.694 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:25:31.733 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:25:32.802 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:27:32.872 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:27:32.916 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:27:33.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:27:34.014 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:27:34.014 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:27:34.015 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:27:34.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:27:34.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:27:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:27:34.495 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:27:34.495 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:27:34.534 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:27:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:27:34.600 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:27:34.601 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:27:34.639 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:27:35.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:29:35.841 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:29:35.879 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:29:36.974 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:29:36.991 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:29:36.991 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:29:36.991 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:29:36.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:29:37.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:29:37.236 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:29:37.479 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:29:37.479 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:29:37.506 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:29:37.548 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:29:37.572 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:29:37.573 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:29:37.611 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:29:38.652 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:31:38.695 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:31:38.733 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:31:39.822 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:31:39.840 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:31:39.840 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:31:39.841 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:31:39.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:31:40.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:31:40.085 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:31:40.319 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:31:40.319 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:31:40.360 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:31:40.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:31:40.425 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:31:40.426 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:31:40.512 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:31:41.537 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:33:41.582 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:33:41.617 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:33:42.586 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:33:42.603 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:33:42.603 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:33:42.603 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:33:42.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:33:42.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:33:42.859 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:33:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:33:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:33:43.119 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:33:43.146 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:33:43.184 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:33:43.186 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:33:43.223 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:33:44.217 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:35:44.263 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:35:44.302 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:35:45.336 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:35:45.353 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:35:45.353 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:35:45.353 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:35:45.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:35:45.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:35:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:35:45.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:35:45.851 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:35:45.880 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:35:45.919 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:35:45.946 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:35:45.948 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:35:45.983 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:35:47.004 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:37:47.050 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:37:47.087 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:37:48.130 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:37:48.149 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:37:48.150 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:37:48.150 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:37:48.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:37:48.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:37:48.392 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:37:48.629 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:37:48.630 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:37:48.668 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:37:48.700 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:37:48.733 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:37:48.735 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:37:48.772 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:37:49.798 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:39:49.874 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:39:49.914 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:39:51.013 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:39:51.030 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:39:51.030 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:39:51.030 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:39:51.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:39:51.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:39:51.287 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:39:51.507 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:39:51.507 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:39:51.548 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:39:51.575 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:39:51.615 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:39:51.616 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:39:51.652 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:39:52.659 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:41:52.703 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:41:52.741 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:41:53.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:41:53.718 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:41:53.718 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:41:53.718 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:41:53.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:41:53.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:41:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:41:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:41:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:41:54.238 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:41:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:41:54.305 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:41:54.306 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:41:54.343 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:41:55.339 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:43:55.386 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:43:55.422 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:43:56.386 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:43:56.401 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:43:56.401 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:43:56.401 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:43:56.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:43:56.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:43:56.644 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:43:56.881 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:43:56.881 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:43:56.920 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:43:56.951 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:43:56.985 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:43:56.987 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:43:57.021 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:43:58.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:45:58.056 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:45:58.095 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:45:59.083 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:45:59.099 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:45:59.099 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:45:59.099 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:45:59.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:45:59.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:45:59.359 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:45:59.583 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:45:59.583 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:45:59.625 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:45:59.651 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:45:59.692 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:45:59.693 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:45:59.731 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:46:00.774 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:48:00.820 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:48:00.859 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:48:01.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:48:01.872 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:48:01.872 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:48:01.872 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:48:01.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:48:02.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:48:02.113 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:48:02.362 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:48:02.362 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:48:02.391 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:48:02.431 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:48:02.456 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:48:02.458 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:48:02.495 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:48:03.517 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:50:03.570 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:50:03.606 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:50:04.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:50:04.719 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:50:04.719 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:50:04.719 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:50:04.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:50:04.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:50:04.958 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:50:05.201 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:50:05.202 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:50:05.243 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:50:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:50:05.310 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:50:05.311 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:50:05.346 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:50:06.364 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:52:06.410 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:52:06.448 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:52:07.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:52:07.510 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:52:07.510 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:52:07.510 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:52:07.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:52:07.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:52:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:52:07.997 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:52:07.997 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:52:08.036 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:52:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:52:08.103 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:52:08.105 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:52:08.142 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:52:09.170 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:54:09.217 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:54:09.260 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:54:10.347 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:54:10.365 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:54:10.365 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:54:10.365 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:54:10.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:54:10.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:54:10.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:54:10.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:54:10.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:54:10.880 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:54:10.919 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:54:10.945 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:54:10.946 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:54:10.985 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:54:12.056 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:56:12.115 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:56:12.154 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:56:13.200 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:56:13.218 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:56:13.218 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:56:13.218 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:56:13.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:56:13.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:56:13.462 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:56:13.694 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:56:13.694 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:56:13.733 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:56:13.762 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:56:13.799 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:56:13.800 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:56:13.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:56:14.922 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:58:14.968 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T23:58:15.011 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:58:16.050 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-19T23:58:16.067 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-19T23:58:16.067 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-19T23:58:16.068 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:58:16.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:58:16.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T23:58:16.327 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-19T23:58:16.540 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-19T23:58:16.540 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-19T23:58:16.578 INFO:teuthology.misc:Compressing logs... 2021-03-19T23:58:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-19T23:58:16.643 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-19T23:58:16.645 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-19T23:58:16.683 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-19T23:58:17.706 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:00:17.757 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:00:17.805 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:00:18.949 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:00:18.971 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:00:18.971 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:00:18.972 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:00:18.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:00:19.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:00:19.218 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:00:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:00:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:00:19.501 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:00:19.542 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:00:19.567 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:00:19.569 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:00:19.613 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:00:20.671 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:02:20.721 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:02:20.761 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:02:21.806 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:02:21.822 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:02:21.822 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:02:21.823 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:02:21.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:02:22.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:02:22.076 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:02:22.312 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:02:22.313 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:02:22.353 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:02:22.383 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:02:22.419 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:02:22.420 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:02:22.457 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:02:23.471 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:04:23.532 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:04:23.570 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:04:24.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:04:24.618 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:04:24.618 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:04:24.618 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:04:24.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:04:24.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:04:24.880 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:04:25.098 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:04:25.098 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:04:25.137 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:04:25.167 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:04:25.202 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:04:25.204 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:04:25.240 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:04:26.248 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:06:26.296 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:06:26.335 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:06:27.420 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:06:27.480 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:06:27.480 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:06:27.480 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:06:27.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:06:27.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:06:27.727 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:06:27.983 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:06:27.983 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:06:28.012 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:06:28.054 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:06:28.080 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:06:28.082 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:06:28.158 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:06:29.247 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:08:29.294 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:08:29.332 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:08:30.368 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:08:30.393 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:08:30.393 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:08:30.393 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:08:30.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:08:30.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:08:30.637 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:08:30.882 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:08:30.882 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:08:30.921 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:08:30.953 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:08:30.988 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:08:30.990 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:08:31.029 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:08:32.049 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:10:32.099 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:10:32.138 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:10:33.183 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:10:33.200 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:10:33.200 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:10:33.200 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:10:33.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:10:33.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:10:33.463 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:10:33.684 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:10:33.684 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:10:33.724 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:10:33.755 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:10:33.789 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:10:33.791 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:10:33.827 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:10:34.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:12:34.884 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:12:34.920 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:12:35.927 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:12:35.947 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:12:35.947 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:12:35.947 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:12:35.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:12:36.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:12:36.188 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:12:36.441 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:12:36.441 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:12:36.471 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:12:36.516 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:12:36.550 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:12:36.551 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:12:36.588 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:12:37.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:14:37.659 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:14:37.699 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:14:38.667 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:14:38.683 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:14:38.684 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:14:38.684 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:14:38.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:14:38.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:14:38.924 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:14:57.218 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T00:14:57.256 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:14:58.266 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:16:58.316 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:16:58.355 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:16:59.410 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:16:59.426 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:16:59.426 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:16:59.426 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:16:59.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:16:59.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:16:59.689 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:17:09.826 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T00:17:09.861 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:17:10.861 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:19:10.910 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:19:10.948 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:19:11.981 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:19:12.002 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:19:12.002 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:19:12.002 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:19:12.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:19:12.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:19:12.248 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:19:12.524 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:19:12.555 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:19:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:19:12.608 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:19:12.608 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:19:12.610 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:19:12.640 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:19:12.672 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:19:12.710 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:19:12.710 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:19:12.739 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:19:12.740 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:19:12.741 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:19:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:19:12.805 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:19:12.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:19:12.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:19:12.874 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:19:12.875 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:19:12.914 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:19:14.069 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:21:14.114 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:21:14.152 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:21:15.178 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:21:15.195 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:21:15.195 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:21:15.195 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:21:15.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:21:15.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:21:15.442 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:21:15.642 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:21:15.674 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:21:15.674 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:21:15.708 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:21:15.708 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:21:15.709 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:21:15.744 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:21:15.813 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:21:15.813 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:21:15.934 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:21:15.966 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:21:15.968 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:21:16.040 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:21:16.040 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:21:16.066 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:21:16.066 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:21:16.106 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:21:17.169 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:23:17.219 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:23:17.259 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:23:18.288 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:23:18.305 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:23:18.305 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:23:18.305 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:23:18.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:23:18.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:23:18.564 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:23:18.795 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:23:18.826 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:23:18.827 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:23:18.862 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:23:18.862 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:23:18.864 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:23:18.897 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:23:18.929 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:23:18.931 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:23:18.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:23:20.061 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:25:20.109 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:25:20.147 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:25:21.146 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:25:21.163 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:25:21.163 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:25:21.163 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:25:21.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:25:21.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:25:21.402 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:25:21.598 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:25:21.630 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:25:21.631 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:25:21.664 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:25:21.664 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:25:21.665 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:25:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:25:21.730 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:25:21.731 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:25:21.769 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:25:22.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:27:22.798 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:27:22.851 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:27:23.918 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:27:23.935 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:27:23.935 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:27:23.935 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:27:23.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:27:24.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:27:24.178 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:27:24.428 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:27:24.428 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:27:24.455 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:27:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:27:24.520 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:27:24.521 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:27:24.558 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:27:25.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:29:25.740 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:29:25.785 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:29:26.833 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:29:26.850 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:29:26.850 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:29:26.851 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:29:26.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:29:27.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:29:27.109 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:29:27.336 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:29:27.336 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:29:27.374 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:29:27.407 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:29:27.442 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:29:27.444 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:29:27.486 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:29:28.432 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:31:28.477 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:31:28.513 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:31:29.511 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:31:29.530 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:31:29.530 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:31:29.530 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:31:29.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:31:29.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:31:29.773 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:31:30.013 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:31:30.013 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:31:30.051 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:31:30.083 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:31:30.117 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T00:31:30.118 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:31:30.157 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:31:31.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:33:31.245 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:33:31.282 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:33:32.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:33:32.378 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:33:32.379 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:33:32.379 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:33:32.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:33:32.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:33:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:33:32.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:33:32.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:33:32.894 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:33:32.927 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:33:32.958 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:33:32.997 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:33:32.997 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:33:33.023 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:33:33.024 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:33:33.025 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:33:33.066 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:33:33.135 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:33:33.135 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:33:34.670 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-20T00:33:34.707 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:33:35.699 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:35:35.751 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:35:35.792 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:35:36.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:35:36.863 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:35:36.863 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:35:36.863 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:35:36.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:35:37.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:35:37.124 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:35:55.458 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T00:35:55.496 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:35:56.477 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:37:56.523 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:37:56.560 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:37:57.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:37:57.630 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:37:57.630 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:37:57.630 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:37:57.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:37:57.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:37:57.877 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:37:57.905 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T00:37:58.964 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T00:38:00.062 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T00:38:01.161 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T00:38:02.261 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T00:38:03.358 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T00:38:04.454 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T00:38:05.553 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T00:38:06.651 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T00:38:07.748 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T00:38:07.748 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T00:38:07.785 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:38:08.913 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:40:08.962 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:40:09.002 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:40:10.045 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:40:10.064 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:40:10.064 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:40:10.064 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:40:10.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:40:10.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:40:10.310 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:40:17.356 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-20T00:40:17.357 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T00:40:17.393 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:40:18.457 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:42:18.505 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:42:18.588 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:42:19.611 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:42:19.627 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:42:19.627 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:42:19.627 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:42:19.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:42:19.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:42:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:42:20.160 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T00:42:20.190 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:42:20.190 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:42:20.218 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:42:20.219 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:42:20.221 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:42:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:42:20.276 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T00:42:20.326 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:42:20.326 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:42:20.341 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:42:20.341 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:42:20.342 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:42:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:42:20.404 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T00:42:20.460 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:42:20.461 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:42:20.476 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:42:20.477 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:42:20.516 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:42:21.516 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:44:21.560 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:44:21.596 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:44:22.622 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:44:22.639 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:44:22.639 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:44:22.639 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:44:22.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:44:22.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:44:22.881 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:45:22.854 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T00:45:22.892 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:45:23.907 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:47:23.954 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:47:23.993 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:47:25.001 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:47:25.018 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:47:25.018 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:47:25.018 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:47:25.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:47:25.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:47:25.276 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:47:25.570 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T00:47:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:47:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:47:25.631 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:47:25.631 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:47:25.633 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:47:25.671 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:47:25.738 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:47:25.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:47:25.773 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:47:25.806 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:47:25.826 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T00:47:25.872 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:47:25.873 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:47:25.887 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:47:25.888 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:47:25.926 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:47:26.926 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:49:27.027 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:49:27.067 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:49:28.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:49:28.039 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:49:28.039 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:49:28.039 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:49:28.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:49:28.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:49:28.282 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:49:28.589 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T00:49:28.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:49:28.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:49:28.651 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:49:28.652 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:49:28.654 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:49:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:49:28.761 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:49:28.761 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:49:28.798 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:49:28.830 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:49:28.850 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T00:49:28.899 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:49:28.899 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:49:28.914 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:49:28.915 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:49:28.948 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:49:29.926 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:51:29.974 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:51:30.013 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:51:31.006 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:51:31.023 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:51:31.023 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:51:31.023 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:51:31.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:51:31.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:51:31.264 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:51:31.592 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:51:31.593 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:51:31.625 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:51:31.665 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:51:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:51:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:51:31.773 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:51:31.808 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:51:31.878 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:51:31.878 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:51:31.931 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:51:32.860 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:53:32.908 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:53:38.226 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:53:40.021 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:53:40.056 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:53:40.056 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:53:40.056 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:53:40.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:53:40.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:53:40.321 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:53:40.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:53:40.637 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:53:40.668 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:53:40.708 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:53:40.779 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:53:40.779 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:53:40.815 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:53:40.848 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:53:40.917 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T00:53:40.917 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:53:40.998 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:53:42.799 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:55:42.847 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:55:42.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:55:43.975 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:55:43.992 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:55:43.992 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:55:43.992 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:55:43.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:55:44.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:55:44.238 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:55:44.578 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:55:44.579 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:55:44.608 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:55:44.649 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:55:44.719 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:55:44.719 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:55:44.759 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:55:44.788 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:55:44.825 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.smithi205.log.gz already exists; not overwritten 2021-03-20T00:55:44.825 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T00:55:44.825 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T00:55:44.833 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:55:44.871 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:55:45.904 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:57:45.951 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:57:45.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:57:47.059 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:57:47.076 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:57:47.076 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:57:47.076 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:57:47.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:57:47.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:57:47.323 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:57:47.672 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:57:47.672 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:57:47.703 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:57:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:57:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:57:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:57:47.859 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:57:47.886 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:57:47.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.smithi205.log.gz already exists; not overwritten 2021-03-20T00:57:47.897 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T00:57:47.897 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T00:57:47.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:57:47.938 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:57:48.990 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:59:49.040 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T00:59:49.077 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:59:50.249 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T00:59:50.268 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T00:59:50.268 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T00:59:50.269 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:59:50.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:59:50.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T00:59:50.529 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T00:59:50.857 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T00:59:50.857 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:59:50.884 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:59:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T00:59:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T00:59:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T00:59:51.033 INFO:teuthology.misc:Compressing logs... 2021-03-20T00:59:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T00:59:51.088 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.smithi205.log.gz already exists; not overwritten 2021-03-20T00:59:51.088 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T00:59:51.088 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T00:59:51.089 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T00:59:51.129 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T00:59:52.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:01:52.289 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:01:52.330 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:01:54.258 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:01:54.279 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:01:54.279 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:01:54.279 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:01:54.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:01:54.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:01:54.534 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:01:54.863 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:01:54.863 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:01:54.895 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:01:54.934 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:01:55.005 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:01:55.006 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:01:55.046 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:01:55.103 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.smithi205.log.gz already exists; not overwritten 2021-03-20T01:01:55.103 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T01:01:55.103 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T01:01:55.109 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:01:55.112 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:01:55.155 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:01:56.343 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:03:56.396 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:03:56.433 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:03:57.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:03:57.906 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:03:57.906 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:03:57.906 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:03:57.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:03:58.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:03:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:03:58.514 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:03:58.514 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:03:58.545 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:03:58.586 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:03:58.656 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:03:58.656 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:03:58.694 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:03:58.725 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:03:58.752 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.smithi205.log.gz already exists; not overwritten 2021-03-20T01:03:58.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T01:03:58.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T01:03:58.755 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:03:58.793 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:03:59.853 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:05:59.902 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:05:59.940 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:06:01.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:06:01.859 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:06:01.859 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:06:01.859 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:06:01.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:06:02.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:06:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:06:02.455 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:06:02.455 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:06:02.484 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:06:02.525 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:06:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:06:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:06:02.634 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:06:02.663 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:06:02.689 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.smithi205.log.gz already exists; not overwritten 2021-03-20T01:06:02.689 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T01:06:02.689 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T01:06:02.691 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:06:02.751 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:06:03.817 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:08:03.862 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:08:03.903 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:08:04.930 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:08:04.947 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:08:04.947 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:08:04.947 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:08:04.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:08:05.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:08:05.197 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:08:05.533 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:08:05.533 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:08:05.566 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:08:05.601 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:08:05.670 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:08:05.670 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:08:05.713 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:08:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:08:05.767 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.smithi205.log.gz already exists; not overwritten 2021-03-20T01:08:05.768 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T01:08:05.768 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T01:08:05.769 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:08:05.808 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:08:06.911 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:10:06.961 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:10:07.000 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:10:08.037 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:10:08.055 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:10:08.055 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:10:08.055 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:10:08.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:10:08.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:10:08.301 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:10:08.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:10:08.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:10:08.669 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:10:08.704 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:10:08.772 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:10:08.773 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:10:08.808 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:10:08.839 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:10:08.864 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.smithi205.log.gz already exists; not overwritten 2021-03-20T01:10:08.864 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T01:10:08.864 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T01:10:08.866 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:10:08.900 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:10:09.898 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:12:09.944 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:12:09.983 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:12:10.990 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:12:11.007 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:12:11.007 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:12:11.007 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:12:11.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:12:11.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:12:11.268 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:12:11.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:12:11.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:12:11.635 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:12:11.673 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:12:11.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:12:11.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:12:11.778 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:12:11.816 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:12:11.836 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.smithi205.log.gz already exists; not overwritten 2021-03-20T01:12:11.837 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T01:12:11.837 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T01:12:11.839 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:12:11.874 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:12:12.903 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:14:12.971 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:14:13.016 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:14:14.043 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:14:14.060 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:14:14.060 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:14:14.060 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:14:14.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:14:14.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:14:14.303 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:14:14.897 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:14:14.935 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:14:14.935 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:14:14.976 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:14:14.976 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:14:14.980 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:14:15.009 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:14:15.092 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:14:15.092 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:14:15.157 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:14:15.186 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:14:15.197 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T01:14:15.255 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:14:15.255 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:14:15.274 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:14:15.275 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:14:15.310 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:14:16.370 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:16:16.430 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:16:16.468 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:16:17.538 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:16:17.555 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:16:17.555 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:16:17.555 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:16:17.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:16:17.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:16:17.799 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:16:36.162 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T01:16:36.255 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:16:37.379 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:18:37.426 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:18:37.467 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:18:38.505 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:18:38.522 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:18:38.522 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:18:38.522 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:18:38.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:18:38.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:18:38.782 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:18:41.794 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T01:18:41.832 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:18:42.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:20:42.881 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:20:42.918 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:20:43.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:20:43.989 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:20:43.989 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:20:43.989 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:20:43.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:20:44.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:20:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:20:44.525 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:20:44.558 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:20:44.558 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:20:44.584 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:20:44.584 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:20:44.586 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:20:44.629 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:20:44.644 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T01:20:44.700 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:20:44.700 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:20:44.716 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:20:44.716 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:20:44.717 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:20:44.771 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:20:44.772 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T01:20:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:20:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:20:44.849 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:20:44.850 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:20:44.888 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:20:45.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:22:45.977 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:22:46.016 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:22:47.111 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:22:47.128 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:22:47.128 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:22:47.129 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:22:47.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:22:47.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:22:47.381 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:23:05.795 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T01:23:05.833 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:23:06.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:25:06.989 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:25:07.036 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:25:08.951 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:25:08.972 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:25:08.972 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:25:08.972 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:25:08.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:25:09.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:25:09.233 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:25:09.526 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:25:09.557 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:25:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:25:09.579 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:25:09.579 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:25:09.581 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:25:09.628 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:25:09.700 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:25:09.701 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:25:09.731 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:25:09.769 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:25:09.790 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T01:25:09.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:25:09.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:25:09.853 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:25:09.853 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:25:09.894 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:25:11.091 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:27:11.159 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:27:11.197 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:27:12.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:27:12.970 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:27:12.970 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:27:12.970 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:27:12.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:27:13.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:27:13.225 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:27:13.522 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:27:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:27:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:27:13.583 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:27:13.583 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:27:13.585 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:27:13.625 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:27:13.695 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:27:13.695 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:27:13.734 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:27:13.764 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:27:13.787 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T01:27:13.834 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:27:13.835 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:27:13.851 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:27:13.851 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:27:13.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:27:14.944 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:29:14.992 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:29:15.032 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:29:16.086 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:29:16.105 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:29:16.106 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:29:16.106 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:29:16.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:29:16.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:29:16.353 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:29:16.657 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:29:16.691 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:29:16.691 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:29:16.716 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:29:16.716 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:29:16.719 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:29:16.762 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:29:16.832 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:29:16.832 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:29:16.869 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:29:16.902 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:29:16.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:29:16.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:29:17.034 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:29:18.048 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:31:18.111 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:31:18.158 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:31:19.301 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:31:19.318 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:31:19.319 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:31:19.319 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:31:19.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:31:19.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:31:19.590 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:31:19.885 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:31:19.917 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:31:19.918 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:31:19.948 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:31:19.949 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:31:19.951 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:31:19.988 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:31:20.059 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:31:20.059 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:31:20.098 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:31:20.131 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:31:20.180 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T01:31:20.182 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:31:20.234 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:31:21.338 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:33:21.385 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:33:21.423 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:33:22.434 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:33:22.450 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:33:22.450 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:33:22.450 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:33:22.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:33:22.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:33:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:33:22.966 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:33:22.998 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:33:22.998 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:33:23.027 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:33:23.027 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:33:23.029 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:33:23.068 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:33:23.138 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:33:23.138 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:33:23.185 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:33:23.219 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:33:23.276 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/562206ea-891b-11eb-909e-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-03-20T01:33:23.276 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/562206ea-891b-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T01:33:23.276 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T01:33:23.278 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:33:23.317 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:33:24.356 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:35:24.402 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:35:24.440 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:35:25.495 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:35:25.523 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:35:25.523 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:35:25.523 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:35:25.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:35:25.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:35:25.770 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:35:26.072 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:35:26.105 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:35:26.105 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:35:26.132 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:35:26.132 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:35:26.136 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:35:26.175 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:35:26.248 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:35:26.248 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:35:26.282 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:35:26.318 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:35:26.340 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/562206ea-891b-11eb-909e-001a4aab830c/ceph-osd.7.log.gz already exists; not overwritten 2021-03-20T01:35:26.341 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/562206ea-891b-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T01:35:26.341 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T01:35:26.341 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:35:26.385 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:35:27.454 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:37:27.507 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:37:27.547 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:37:28.526 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:37:28.542 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:37:28.543 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:37:28.543 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:37:28.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:37:28.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:37:28.800 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:37:29.081 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:37:29.112 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:37:29.112 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:37:29.141 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:37:29.141 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:37:29.143 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:37:29.182 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:37:29.250 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:37:29.251 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:37:29.290 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:37:29.319 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:37:29.345 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/562206ea-891b-11eb-909e-001a4aab830c/ceph-osd.7.log.gz already exists; not overwritten 2021-03-20T01:37:29.345 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/562206ea-891b-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T01:37:29.345 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T01:37:29.346 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:37:29.378 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:37:30.371 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:39:30.423 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:39:30.462 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:39:31.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:39:31.512 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:39:31.512 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:39:31.512 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:39:31.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:39:31.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:39:31.762 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:39:32.071 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:39:32.103 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:39:32.103 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:39:32.130 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:39:32.131 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:39:32.131 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:39:32.173 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:39:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:39:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:39:32.288 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:39:32.315 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:39:32.344 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/562206ea-891b-11eb-909e-001a4aab830c/ceph-osd.7.log.gz already exists; not overwritten 2021-03-20T01:39:32.345 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/562206ea-891b-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T01:39:32.345 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T01:39:32.346 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:39:32.384 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:39:33.426 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:41:33.472 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:41:33.512 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:41:34.544 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:41:34.561 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:41:34.561 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:41:34.561 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:41:34.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:41:34.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:41:34.807 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:41:37.858 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T01:41:37.896 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:41:38.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:43:39.014 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:43:39.055 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:43:40.205 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:43:40.225 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:43:40.225 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:43:40.226 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:43:40.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:43:40.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:43:40.494 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:43:40.547 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T01:43:40.616 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:43:41.764 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:45:41.812 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:45:41.851 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:45:42.911 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:45:42.928 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:45:42.929 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:45:42.929 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:45:42.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:45:43.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:45:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:45:43.464 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:45:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:45:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:45:43.522 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:45:43.522 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:45:43.524 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:45:43.567 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:45:43.579 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T01:45:43.638 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:45:43.638 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:45:43.656 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:45:43.657 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:45:43.659 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:45:43.710 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:45:43.715 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T01:45:43.780 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:45:43.780 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:45:43.797 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:45:43.797 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:45:43.836 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:45:44.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:47:44.927 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:47:44.966 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:47:45.971 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:47:45.988 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:47:45.988 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:47:45.988 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:47:45.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:47:46.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:47:46.232 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:48:04.674 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T01:48:04.716 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:48:05.749 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:50:05.825 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:50:05.863 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:50:06.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:50:06.994 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:50:06.994 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:50:06.994 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:50:06.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:50:07.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:50:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:50:07.533 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:50:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:50:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:50:07.592 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:50:07.592 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:50:07.593 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:50:07.635 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:50:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:50:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:50:07.738 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:50:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:50:07.790 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T01:50:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:50:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:50:07.962 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:50:07.962 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:50:08.006 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:50:09.113 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:52:09.163 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:52:09.204 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:52:11.032 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:52:11.050 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:52:11.050 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:52:11.050 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:52:11.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:52:11.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:52:11.297 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:52:11.602 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:52:11.634 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:52:11.634 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:52:11.660 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:52:11.660 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:52:11.662 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:52:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:52:11.774 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:52:11.774 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:52:11.813 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:52:11.843 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:52:11.869 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T01:52:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:52:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:52:11.926 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:52:11.926 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:52:11.968 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:52:13.706 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:54:13.755 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:54:13.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:54:15.630 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:54:15.646 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:54:15.647 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:54:15.647 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:54:15.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:54:15.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:54:15.888 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:54:16.196 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:54:16.229 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:54:16.229 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:54:16.254 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:54:16.254 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:54:16.257 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:54:16.298 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:54:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:54:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:54:16.421 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:54:16.513 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T01:54:16.524 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:54:16.595 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:54:16.595 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:54:16.613 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:54:16.613 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:54:16.651 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:54:17.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:56:17.871 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:56:17.910 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:56:19.880 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:56:19.897 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:56:19.897 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:56:19.897 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:56:19.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:56:20.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:56:20.157 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:56:20.444 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:56:20.475 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:56:20.475 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:56:20.507 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:56:20.507 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:56:20.509 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:56:20.544 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:56:20.612 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:56:20.613 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:56:20.654 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:56:20.680 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:56:20.761 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T01:56:20.761 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:56:20.860 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:56:22.718 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:58:22.771 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T01:58:22.817 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:58:24.682 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T01:58:24.700 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T01:58:24.700 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T01:58:24.700 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:58:24.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:58:24.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T01:58:24.946 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T01:58:25.230 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T01:58:25.262 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:58:25.262 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:58:25.281 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T01:58:25.281 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:58:25.283 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:58:25.332 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T01:58:25.403 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T01:58:25.403 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T01:58:25.448 INFO:teuthology.misc:Compressing logs... 2021-03-20T01:58:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T01:58:25.482 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T01:58:25.504 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/eafbe59e-891e-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T01:58:25.506 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T01:58:25.545 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T01:58:26.626 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:00:26.673 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:00:26.722 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:00:27.919 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:00:27.942 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:00:27.942 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:00:27.943 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:00:27.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:00:28.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:00:28.201 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:00:28.490 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T02:00:28.522 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:00:28.522 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:00:28.551 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:00:28.551 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:00:28.554 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:00:28.593 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:00:28.663 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:00:28.663 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:00:28.702 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:00:28.732 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:00:28.745 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T02:00:28.768 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/eafbe59e-891e-11eb-909e-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2021-03-20T02:00:28.787 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/eafbe59e-891e-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T02:00:28.789 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T02:00:28.838 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:00:29.985 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:02:30.034 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:02:30.078 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:02:32.293 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:02:32.311 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:02:32.312 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:02:32.312 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:02:32.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:02:32.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:02:32.590 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:02:32.891 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T02:02:32.923 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:02:32.923 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:02:32.950 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:02:32.951 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:02:32.953 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:02:32.994 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:02:33.063 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:02:33.064 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:02:33.101 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:02:33.139 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:02:33.155 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T02:02:33.155 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/eafbe59e-891e-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T02:02:33.157 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T02:02:33.211 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:02:35.427 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:04:35.478 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:04:35.518 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:04:37.475 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:04:37.494 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:04:37.494 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:04:37.494 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:04:37.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:04:37.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:04:37.746 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:04:38.043 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T02:04:38.077 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:04:38.077 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:04:38.102 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:04:38.102 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:04:38.104 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:04:38.149 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:04:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:04:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:04:38.253 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:04:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:04:38.309 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T02:04:38.309 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/eafbe59e-891e-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T02:04:38.311 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T02:04:38.350 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:04:39.532 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:06:39.587 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:06:39.629 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:06:41.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:06:41.607 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:06:41.607 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:06:41.608 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:06:41.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:06:41.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:06:41.858 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:07:00.290 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T02:07:00.330 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:07:01.832 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:09:01.884 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:09:01.938 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:09:04.181 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:09:04.202 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:09:04.202 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:09:04.202 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:09:04.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:09:04.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:09:04.480 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:09:07.490 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T02:09:07.530 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:09:09.351 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:11:09.412 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:11:09.474 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:11:11.621 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:11:11.639 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:11:11.639 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:11:11.639 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:11:11.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:11:11.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:11:11.892 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:11:14.370 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T02:11:14.410 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:11:15.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:13:15.580 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:13:15.665 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:13:16.840 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:13:16.862 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:13:16.862 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:13:16.863 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:13:16.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:13:17.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:13:17.125 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:13:17.414 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T02:13:17.448 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:13:17.448 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:13:17.478 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:13:17.478 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:13:17.479 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:13:17.519 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:13:17.534 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T02:13:17.590 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:13:17.590 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:13:17.607 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:13:17.607 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:13:17.609 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:13:17.661 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:13:17.666 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T02:13:17.729 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:13:17.729 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:13:17.746 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:13:17.746 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:13:17.786 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:13:18.851 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:15:18.897 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:15:19.391 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:15:20.515 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:15:20.532 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:15:20.533 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:15:20.533 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:15:20.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:15:20.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:15:20.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:15:21.059 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T02:15:21.092 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:15:21.092 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:15:21.116 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:15:21.117 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:15:21.118 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:15:21.163 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:15:21.172 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T02:15:21.232 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:15:21.233 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:15:21.247 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:15:21.247 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:15:21.249 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:15:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:15:21.303 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T02:15:21.371 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:15:21.371 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:15:21.387 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:15:21.387 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:15:21.463 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:15:22.602 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:17:22.648 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:17:22.688 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:17:23.806 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:17:23.823 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:17:23.823 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:17:23.823 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:17:23.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:17:24.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:17:24.069 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:17:24.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T02:17:24.522 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:17:25.545 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:19:25.592 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:19:25.668 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:19:27.598 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:19:27.615 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:19:27.615 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:19:27.615 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:19:27.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:19:27.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:19:27.862 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:19:28.164 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T02:19:28.195 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:19:28.195 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:19:28.223 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:19:28.223 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:19:28.225 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:19:28.266 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:19:28.340 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:19:28.341 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:19:28.372 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:19:28.409 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:19:28.417 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T02:19:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:19:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:19:28.495 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:19:28.495 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:19:28.535 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:19:29.586 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:21:29.636 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:21:29.677 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:21:30.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:21:30.704 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:21:30.704 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:21:30.705 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:21:30.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:21:30.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:21:30.965 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:21:31.271 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:21:31.271 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:21:31.300 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:21:31.342 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:21:31.411 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:21:31.411 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:21:31.452 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:21:31.479 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:21:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:21:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:21:31.604 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:21:32.638 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:23:32.687 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:23:32.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:23:33.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:23:33.833 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:23:33.833 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:23:33.834 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:23:33.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:23:34.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:23:34.087 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:23:34.430 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:23:34.430 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:23:34.458 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:23:34.500 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:23:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:23:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:23:34.615 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:23:34.641 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:23:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:23:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:23:34.782 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:23:35.854 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:25:35.905 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:25:35.944 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:25:37.001 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:25:37.017 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:25:37.017 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:25:37.017 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:25:37.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:25:37.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:25:37.260 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:25:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:25:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:25:37.619 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:25:37.660 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:25:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:25:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:25:37.767 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:25:37.799 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:25:37.868 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:25:37.868 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:25:37.929 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:25:38.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:27:39.049 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:27:39.094 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:27:40.172 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:27:40.189 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:27:40.189 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:27:40.189 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:27:40.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:27:40.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:27:40.447 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:27:40.767 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:27:40.767 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:27:40.799 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:27:40.835 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:27:40.905 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:27:40.905 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:27:40.944 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:27:40.973 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:27:41.028 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T02:27:41.029 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T02:27:41.066 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:27:42.098 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:29:42.170 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:29:42.269 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:29:43.350 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:29:43.369 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:29:43.369 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:29:43.370 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:29:43.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:29:43.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:29:43.619 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:29:43.951 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:29:43.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:29:43.985 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:29:44.022 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:29:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:29:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:29:44.125 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:29:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:29:44.192 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T02:29:44.193 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T02:29:44.195 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T02:29:44.234 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:29:45.304 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:31:45.365 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:31:45.406 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:31:46.452 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:31:46.469 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:31:46.469 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:31:46.469 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:31:46.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:31:46.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:31:46.717 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:31:47.057 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:31:47.057 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:31:47.087 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:31:47.128 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:31:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:31:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:31:47.242 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:31:47.270 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:31:47.326 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T02:31:47.326 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T02:31:47.345 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T02:31:47.347 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T02:31:47.388 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:31:48.465 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:33:48.529 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:33:48.602 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:33:49.645 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:33:49.661 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:33:49.662 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:33:49.662 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:33:49.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:33:49.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:33:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:33:50.249 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:33:50.249 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:33:50.280 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:33:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:33:50.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:33:50.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:33:50.435 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:33:50.462 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:33:50.491 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T02:33:50.492 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T02:33:50.492 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T02:33:50.494 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T02:33:50.531 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:33:51.630 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:35:51.679 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:35:51.721 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:35:52.741 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:35:52.759 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:35:52.759 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:35:52.759 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:35:52.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:35:52.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:35:53.006 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:35:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:35:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:35:53.384 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:35:53.422 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:35:53.490 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:35:53.490 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:35:53.529 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:35:53.557 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:35:53.582 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T02:35:53.582 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T02:35:53.582 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T02:35:53.583 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T02:35:53.583 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T02:35:53.583 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T02:35:53.584 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T02:35:53.621 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:35:54.711 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:37:54.759 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:37:54.797 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:37:55.851 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:37:55.869 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:37:55.870 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:37:55.870 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:37:55.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:37:56.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:37:56.119 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:37:56.462 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:37:56.462 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:37:56.492 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:37:56.531 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:37:56.600 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:37:56.600 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:37:56.640 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:37:56.669 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:37:56.696 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T02:37:56.696 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T02:37:56.697 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T02:37:56.697 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T02:37:56.697 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T02:37:56.698 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T02:37:56.700 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T02:37:56.739 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:37:57.799 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:39:57.851 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:39:57.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:39:58.945 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:39:58.965 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:39:58.965 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:39:58.965 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:39:58.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:39:59.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:39:59.226 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:39:59.561 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:39:59.561 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:39:59.593 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:39:59.633 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:39:59.704 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:39:59.704 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:39:59.745 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:39:59.772 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:39:59.803 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T02:39:59.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T02:39:59.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T02:39:59.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T02:39:59.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T02:39:59.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T02:39:59.806 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T02:39:59.858 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:40:00.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:42:00.949 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:42:00.988 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:42:02.113 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:42:02.130 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:42:02.130 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:42:02.130 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:42:02.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:42:02.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:42:02.376 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:42:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:42:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:42:02.748 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:42:02.788 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:42:02.858 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:42:02.859 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:42:02.898 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:42:02.928 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:42:02.954 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T02:42:02.955 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T02:42:02.955 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T02:42:02.955 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T02:42:02.956 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/2a8cf190-8923-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T02:42:02.956 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T02:42:02.957 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T02:42:03.013 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:42:04.073 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:44:04.120 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:44:04.158 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:44:05.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:44:05.300 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:44:05.300 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:44:05.300 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:44:05.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:44:05.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:44:05.545 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:44:23.874 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T02:44:23.915 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:44:24.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:46:25.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:46:25.084 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:46:26.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:46:26.187 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:46:26.187 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:46:26.187 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:46:26.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:46:26.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:46:26.449 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:46:29.474 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T02:46:29.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:46:30.585 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:48:30.634 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:48:30.673 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:48:31.740 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:48:31.759 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:48:31.759 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:48:31.759 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:48:31.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:48:31.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:48:32.001 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:48:35.042 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T02:48:35.081 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:48:36.149 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:50:36.215 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:50:36.253 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:50:37.334 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:50:37.351 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:50:37.352 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:50:37.352 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:50:37.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:50:37.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:50:37.601 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:50:38.123 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T02:50:38.156 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:50:38.156 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:50:38.184 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:50:38.185 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:50:38.185 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:50:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:50:38.242 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T02:50:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:50:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:50:38.311 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:50:38.312 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:50:38.313 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:50:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:50:38.371 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T02:50:38.436 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:50:38.436 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:50:38.455 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:50:38.455 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:50:38.496 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:50:39.539 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:52:39.586 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:52:39.626 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:52:40.668 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:52:40.685 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:52:40.685 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:52:40.685 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:52:40.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:52:40.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:52:40.943 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:53:40.901 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T02:53:40.940 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:53:42.020 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:55:42.070 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:55:42.116 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:55:43.313 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:55:43.330 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:55:43.330 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:55:43.331 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:55:43.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:55:43.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:55:43.577 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:55:43.875 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T02:55:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:55:43.907 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:55:43.932 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:55:43.934 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:55:43.935 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:55:43.976 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:55:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:55:44.048 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:55:44.082 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:55:44.116 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:55:44.121 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T02:55:44.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:55:44.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:55:44.200 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:55:44.200 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:55:44.247 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:55:45.494 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:57:45.549 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:57:45.589 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:57:47.823 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:57:47.840 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:57:47.841 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:57:47.841 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:57:47.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:57:48.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:57:48.105 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:57:48.376 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T02:57:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:57:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:57:48.434 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:57:48.434 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:57:48.436 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:57:48.478 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:57:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:57:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:57:48.587 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:57:48.618 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:57:48.638 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T02:57:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:57:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:57:48.703 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:57:48.703 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:57:48.748 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:57:50.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:59:50.291 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T02:59:50.329 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:59:51.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T02:59:51.561 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T02:59:51.562 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T02:59:51.562 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:59:51.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:59:51.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T02:59:51.810 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T02:59:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T02:59:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:59:52.182 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:59:52.221 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T02:59:52.293 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T02:59:52.293 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:59:52.339 INFO:teuthology.misc:Compressing logs... 2021-03-20T02:59:52.366 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T02:59:52.436 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T02:59:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T02:59:52.503 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T02:59:53.594 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:01:53.640 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:01:53.678 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:01:55.488 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:01:55.506 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:01:55.506 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:01:55.506 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:01:55.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:01:55.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:01:55.750 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:01:56.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:01:56.092 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:01:56.123 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:01:56.162 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:01:56.232 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:01:56.232 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:01:56.274 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:01:56.301 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:01:56.362 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:01:56.447 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:01:56.447 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:01:56.449 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:01:56.488 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:01:57.575 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:03:57.626 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:03:57.664 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:03:58.676 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:03:58.694 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:03:58.695 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:03:58.695 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:03:58.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:03:58.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:03:58.955 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:03:59.277 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:03:59.277 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:03:59.309 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:03:59.347 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:03:59.416 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:03:59.417 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:03:59.465 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:03:59.492 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:03:59.520 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:03:59.521 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:03:59.521 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:03:59.521 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:03:59.521 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:03:59.521 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:03:59.521 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:03:59.522 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:03:59.558 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:04:00.562 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:06:00.608 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:06:00.702 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:06:01.740 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:06:01.756 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:06:01.757 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:06:01.757 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:06:01.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:06:01.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:06:02.006 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:06:02.349 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:06:02.349 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:06:02.379 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:06:02.419 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:06:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:06:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:06:02.528 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:06:02.558 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:06:02.584 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:06:02.584 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:06:02.584 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:06:02.584 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:06:02.584 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:06:02.584 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:06:02.584 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:06:02.585 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:06:02.585 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:06:02.623 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:06:03.773 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:08:03.824 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:08:03.862 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:08:04.898 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:08:04.916 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:08:04.916 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:08:04.916 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:08:04.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:08:05.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:08:05.167 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:08:05.498 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:08:05.498 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:08:05.530 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:08:05.569 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:08:05.639 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:08:05.640 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:08:05.677 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:08:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:08:05.732 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:08:05.732 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:08:05.732 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:08:05.732 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:08:05.732 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:08:05.733 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:08:05.733 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:08:05.733 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:08:05.734 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:08:05.772 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:08:06.787 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:10:06.833 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:10:06.875 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:10:07.947 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:10:07.966 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:10:07.966 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:10:07.967 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:10:07.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:10:08.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:10:08.225 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:10:08.567 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:10:08.567 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:10:08.596 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:10:08.637 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:10:08.706 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:10:08.706 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:10:08.748 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:10:08.774 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:10:08.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:10:08.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:10:08.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:10:08.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:10:08.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:10:08.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:10:08.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:10:08.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:10:08.805 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:10:08.852 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:10:09.872 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:12:09.916 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:12:09.956 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:12:11.006 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:12:11.042 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:12:11.042 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:12:11.042 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:12:11.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:12:11.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:12:11.302 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:12:11.697 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:12:11.698 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:12:11.728 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:12:11.768 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:12:11.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:12:11.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:12:11.887 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:12:11.917 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:12:11.920 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:12:11.920 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:12:11.920 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:12:11.920 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:12:11.920 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:12:11.921 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:12:11.921 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:12:11.921 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:12:11.922 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:12:11.960 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:12:13.025 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:14:13.076 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:14:13.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:14:14.177 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:14:14.194 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:14:14.195 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:14:14.195 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:14:14.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:14:14.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:14:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:14:14.778 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:14:14.779 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:14:14.808 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:14:14.852 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:14:14.922 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:14:14.923 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:14:14.962 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:14:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:14:15.015 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:14:15.015 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:14:15.015 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:14:15.015 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:14:15.015 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:14:15.015 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:14:15.015 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:14:15.016 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:14:15.017 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:14:15.056 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:14:16.113 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:16:16.164 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:16:16.218 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:16:17.327 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:16:17.344 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:16:17.344 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:16:17.344 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:16:17.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:16:17.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:16:17.609 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:16:17.934 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:16:17.934 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:16:17.965 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:16:18.003 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:16:18.073 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:16:18.073 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:16:18.113 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:16:18.141 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:16:18.169 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:16:18.169 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:16:18.169 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:16:18.169 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:16:18.169 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:16:18.170 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:16:18.170 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:16:18.170 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:16:18.171 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:16:18.209 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:16:19.298 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:18:19.349 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:18:19.388 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:18:20.399 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:18:20.416 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:18:20.416 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:18:20.417 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:18:20.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:18:20.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:18:20.661 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:18:20.984 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:18:20.985 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:18:21.011 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:18:21.056 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:18:21.126 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:18:21.126 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:18:21.166 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:18:21.195 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:18:21.222 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:18:21.222 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:18:21.222 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:18:21.223 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:18:21.223 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:18:21.223 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:18:21.223 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:18:21.223 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:18:21.224 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:18:21.260 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:18:22.260 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:20:22.304 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:20:22.346 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:20:23.447 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:20:23.466 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:20:23.467 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:20:23.467 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:20:23.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:20:23.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:20:23.722 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:20:24.079 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:20:24.079 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:20:24.108 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:20:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:20:24.219 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:20:24.220 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:20:24.263 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:20:24.290 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:20:24.315 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:20:24.316 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:20:24.316 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:20:24.316 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:20:24.316 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:20:24.316 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:20:24.316 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:20:24.316 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:20:24.317 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:20:24.355 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:20:25.451 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:22:25.507 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:22:25.547 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:22:26.640 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:22:26.659 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:22:26.659 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:22:26.659 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:22:26.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:22:26.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:22:26.926 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:22:27.242 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:22:27.243 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:22:27.272 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:22:27.312 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:22:27.382 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:22:27.382 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:22:27.420 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:22:27.450 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:22:27.478 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:22:27.478 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:22:27.478 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:22:27.478 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:22:27.479 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:22:27.479 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:22:27.479 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:22:27.479 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:22:27.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:22:27.525 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:22:28.741 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:24:28.791 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:24:28.829 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:24:29.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:24:29.886 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:24:29.886 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:24:29.887 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:24:29.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:24:30.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:24:30.131 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:24:30.473 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:24:30.474 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:24:30.502 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:24:30.544 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:24:30.614 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:24:30.614 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:24:30.650 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:24:30.682 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:24:30.703 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:24:30.703 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:24:30.703 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:24:30.704 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:24:30.704 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:24:30.704 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:24:30.704 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:24:30.704 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:24:30.705 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:24:30.744 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:24:31.757 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:26:31.805 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:26:31.849 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:26:32.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:26:32.886 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:26:32.886 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:26:32.886 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:26:32.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:26:33.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:26:33.130 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:26:33.464 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:26:33.464 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:26:33.497 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:26:33.533 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:26:33.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:26:33.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:26:33.636 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:26:33.669 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:26:33.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:26:33.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:26:33.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:26:33.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:26:33.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:26:33.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:26:33.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:26:33.694 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:26:33.697 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:26:33.740 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:26:34.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:28:34.864 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:28:34.903 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:28:35.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:28:35.993 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:28:35.993 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:28:35.993 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:28:35.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:28:36.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:28:36.254 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:28:36.579 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:28:36.579 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:28:36.613 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:28:36.648 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:28:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:28:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:28:36.754 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:28:36.784 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:28:36.797 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:28:36.798 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:28:36.798 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:28:36.798 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:28:36.798 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:28:36.798 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:28:36.799 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:28:36.799 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:28:36.800 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:28:36.842 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:28:37.848 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:30:37.894 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:30:37.944 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:30:39.032 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:30:39.049 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:30:39.050 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:30:39.050 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:30:39.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:30:39.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:30:39.292 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:30:39.613 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:30:39.613 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:30:39.645 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:30:39.683 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:30:39.752 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:30:39.753 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:30:39.790 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:30:39.821 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:30:39.845 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:30:39.845 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:30:39.845 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:30:39.845 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:30:39.845 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:30:39.845 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:30:39.845 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:30:39.846 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:30:39.847 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:30:39.886 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:30:40.994 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:32:41.042 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:32:41.078 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:32:42.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:32:42.209 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:32:42.209 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:32:42.210 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:32:42.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:32:42.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:32:42.461 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:32:42.793 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:32:42.793 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:32:42.827 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:32:42.864 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:32:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:32:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:32:42.973 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:32:43.004 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:32:43.029 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:32:43.030 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:32:43.030 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:32:43.030 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:32:43.030 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:32:43.030 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:32:43.030 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:32:43.030 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:32:43.031 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:32:43.071 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:32:44.164 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:34:44.212 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:34:44.583 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:34:46.822 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:34:46.839 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:34:46.839 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:34:46.840 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:34:46.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:34:47.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:34:47.142 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:34:47.430 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:34:47.430 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:34:47.455 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:34:47.498 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:34:47.566 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:34:47.566 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:34:47.607 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:34:47.632 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:34:47.666 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:34:47.666 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:34:47.666 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:34:47.666 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:34:47.667 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:34:47.667 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:34:47.667 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:34:47.667 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:34:47.668 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:34:47.709 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:34:48.804 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:36:48.849 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:36:48.924 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:36:51.021 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:36:51.040 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:36:51.040 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:36:51.040 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:36:51.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:36:51.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:36:51.289 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:36:51.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:36:51.637 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:36:51.666 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:36:51.707 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:36:51.776 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:36:51.777 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:36:51.808 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:36:51.846 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:36:51.868 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:36:51.868 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:36:51.868 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:36:51.868 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:36:51.868 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:36:51.868 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:36:51.868 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:36:51.868 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:36:51.870 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:36:51.919 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:36:53.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:38:53.844 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:38:53.883 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:38:54.982 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:38:55.001 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:38:55.001 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:38:55.001 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:38:55.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:38:55.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:38:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:38:55.590 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:38:55.591 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:38:55.622 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:38:55.662 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:38:55.731 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:38:55.731 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:38:55.766 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:38:55.800 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:38:55.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:38:55.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:38:55.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:38:55.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:38:55.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:38:55.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:38:55.822 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:38:55.822 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:38:55.823 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:38:55.904 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:38:56.971 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:40:57.022 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:40:57.059 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:40:58.088 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:40:58.106 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:40:58.106 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:40:58.107 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:40:58.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:40:58.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:40:58.368 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:40:58.670 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:40:58.671 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:40:58.701 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:40:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:40:58.811 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:40:58.811 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:40:58.848 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:40:58.878 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:40:58.903 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:40:58.903 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:40:58.904 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:40:58.904 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:40:58.904 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:40:58.904 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:40:58.904 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:40:58.904 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:40:58.905 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:40:58.942 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:41:00.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:43:00.119 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:43:00.159 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:43:02.210 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:43:02.228 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:43:02.228 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:43:02.228 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:43:02.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:43:02.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:43:02.474 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:43:02.838 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:43:02.838 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:43:02.863 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:43:02.910 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:43:02.982 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:43:02.982 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:43:03.023 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:43:03.053 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:43:03.074 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:43:03.074 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:43:03.074 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:43:03.074 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:43:03.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:43:03.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:43:03.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:43:03.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:43:03.075 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:43:03.118 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:43:04.196 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:45:04.245 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:45:04.284 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:45:06.372 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:45:06.403 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:45:06.404 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:45:06.404 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:45:06.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:45:06.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:45:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:45:07.104 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:45:07.104 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:45:07.133 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:45:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:45:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:45:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:45:07.292 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:45:07.321 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:45:07.349 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:45:07.350 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:45:07.350 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:45:07.350 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:45:07.350 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:45:07.351 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:45:07.351 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:45:07.351 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:45:07.353 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:45:07.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:45:08.518 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:47:08.566 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:47:08.608 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:47:10.677 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:47:10.697 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:47:10.697 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:47:10.697 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:47:10.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:47:10.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:47:10.972 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:47:11.312 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:47:11.312 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:47:11.340 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:47:11.384 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:47:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:47:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:47:11.487 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:47:11.521 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:47:11.545 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:47:11.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:47:11.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:47:11.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:47:11.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:47:11.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:47:11.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:47:11.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:47:11.547 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:47:11.587 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:47:12.779 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:49:12.827 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:49:12.868 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:49:14.858 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:49:14.876 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:49:14.876 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:49:14.876 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:49:14.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:49:15.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:49:15.127 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:49:15.471 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:49:15.471 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:49:15.502 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:49:15.542 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:49:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:49:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:49:15.652 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:49:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:49:15.709 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2021-03-20T03:49:15.710 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2021-03-20T03:49:15.710 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-20T03:49:15.710 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T03:49:15.710 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-20T03:49:15.710 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T03:49:15.710 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-20T03:49:15.711 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2021-03-20T03:49:15.712 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T03:49:15.766 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:49:16.962 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:51:17.013 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:51:17.051 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:51:18.162 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:51:18.179 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:51:18.179 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:51:18.180 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:51:18.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:51:18.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:51:18.425 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:51:35.106 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T03:51:35.145 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:51:36.251 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:53:36.303 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:53:36.343 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:53:37.431 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:53:37.448 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:53:37.449 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:53:37.449 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:53:37.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:53:37.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:53:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:53:37.779 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T03:54:38.781 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T03:54:38.819 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:54:39.941 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:56:39.994 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:56:40.034 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:56:41.138 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:56:41.160 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:56:41.160 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:56:41.161 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:56:41.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:56:41.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:56:41.406 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:56:41.614 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:56:41.645 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:56:41.645 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:56:41.691 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T03:56:41.691 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:56:41.693 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:56:41.722 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:56:41.757 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T03:56:41.791 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:56:41.791 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:56:41.824 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T03:56:41.824 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:56:41.825 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:56:41.860 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:56:41.889 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T03:56:41.929 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T03:56:41.929 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:56:41.954 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T03:56:41.954 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T03:56:41.997 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:56:43.049 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:58:43.114 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T03:58:43.151 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:58:44.209 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T03:58:44.225 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T03:58:44.225 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T03:58:44.226 INFO:teuthology.misc:Compressing logs... 2021-03-20T03:58:44.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:58:44.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T03:58:44.491 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T03:58:47.522 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T03:58:47.583 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T03:58:48.720 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:00:48.771 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:00:48.812 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:00:49.918 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:00:49.942 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:00:49.942 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:00:49.942 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:00:49.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:00:50.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:00:50.193 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:00:50.403 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:00:50.434 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:00:50.435 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:00:50.508 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:00:50.508 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:00:50.509 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:00:50.538 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:00:50.609 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:00:50.609 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:00:50.661 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:00:50.697 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:00:50.721 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:00:50.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:00:50.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:00:50.795 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:00:50.795 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:00:50.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:00:51.962 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:02:52.011 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:02:52.049 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:02:53.136 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:02:53.152 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:02:53.153 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:02:53.153 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:02:53.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:02:53.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:02:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:02:53.611 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:02:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:02:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:02:53.678 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:02:53.679 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:02:53.680 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:02:53.714 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:02:53.747 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T04:02:53.749 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T04:02:53.789 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:02:54.833 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:04:54.884 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:04:54.924 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:04:55.978 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:04:55.997 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:04:55.997 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:04:55.997 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:04:55.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:04:56.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:04:56.255 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:04:56.471 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:04:56.471 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:04:56.538 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:04:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:04:56.567 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T04:04:56.569 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T04:04:56.610 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:04:57.630 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:06:57.680 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:06:57.732 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:06:58.829 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:06:58.847 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:06:58.847 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:06:58.848 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:06:58.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:06:59.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:06:59.091 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:06:59.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:06:59.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:06:59.367 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:06:59.395 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:06:59.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T04:06:59.433 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T04:06:59.471 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:07:00.581 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:09:00.642 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:09:00.681 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:09:01.833 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:09:01.851 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:09:01.852 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:09:01.852 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:09:01.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:09:02.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:09:02.096 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:09:02.340 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:09:02.341 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:09:02.379 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:09:02.410 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:09:02.445 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T04:09:02.446 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T04:09:02.484 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:09:03.552 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:11:03.601 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:11:03.643 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:11:04.676 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:11:04.694 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:11:04.694 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:11:04.694 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:11:04.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:11:04.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:11:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:11:05.161 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:11:05.161 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:11:05.207 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:11:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:11:05.272 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T04:11:05.273 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T04:11:05.310 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:11:06.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:13:06.401 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:13:06.441 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:13:07.478 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:13:07.495 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:13:07.495 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:13:07.495 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:13:07.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:13:07.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:13:07.737 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:13:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:13:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:13:08.007 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:13:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:13:08.072 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T04:13:08.073 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T04:13:08.111 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:13:09.186 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:15:09.234 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:15:09.274 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:15:10.389 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:15:10.405 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:15:10.406 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:15:10.406 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:15:10.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:15:10.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:15:10.649 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:15:10.875 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:15:10.875 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:15:10.913 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:15:10.945 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:15:10.979 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T04:15:10.980 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T04:15:11.074 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:15:12.152 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:17:12.199 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:17:12.243 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:17:13.317 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:17:13.335 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:17:13.335 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:17:13.335 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:17:13.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:17:13.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:17:13.596 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:17:13.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:17:13.823 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:17:13.861 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:17:13.891 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:17:13.927 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T04:17:13.929 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T04:17:13.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:17:15.051 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:19:15.109 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:19:15.148 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:19:16.258 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:19:16.275 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:19:16.275 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:19:16.276 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:19:16.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:19:16.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:19:16.520 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:19:16.760 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:19:16.761 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:19:16.800 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:19:16.832 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:19:16.869 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T04:19:16.869 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T04:19:16.918 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:19:18.013 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:21:18.060 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:21:18.101 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:21:19.236 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:21:19.255 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:21:19.255 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:21:19.256 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:21:19.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:21:19.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:21:19.504 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:21:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:21:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:21:19.771 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:21:19.802 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:21:19.835 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T04:21:19.836 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T04:21:19.877 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:21:21.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:23:21.073 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:23:21.112 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:23:22.220 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:23:22.236 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:23:22.237 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:23:22.237 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:23:22.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:23:22.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:23:22.495 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:23:25.506 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T04:23:25.544 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:23:26.601 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:25:26.653 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:25:26.696 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:25:27.851 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:25:27.868 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:25:27.868 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:25:27.868 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:25:27.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:25:28.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:25:28.115 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:25:30.690 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T04:25:30.728 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:25:31.782 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:27:31.830 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:27:31.869 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:27:32.900 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:27:32.917 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:27:32.917 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:27:32.917 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:27:32.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:27:33.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:27:33.161 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:27:33.437 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T04:27:33.467 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:27:33.467 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:27:33.497 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:27:33.497 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:27:33.499 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:27:33.537 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:27:33.552 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T04:27:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:27:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:27:33.618 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:27:33.618 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:27:33.621 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:27:33.674 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:27:33.677 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T04:27:33.741 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:27:33.741 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:27:33.757 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:27:33.757 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:27:33.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:27:34.829 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:29:34.878 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:29:34.916 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:29:35.983 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:29:36.001 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:29:36.001 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:29:36.001 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:29:36.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:29:36.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:29:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:30:08.357 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T04:30:08.394 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:30:08.394 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:30:08.417 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:30:08.417 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:30:08.419 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:30:08.464 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:30:08.474 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T04:30:08.533 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:30:08.533 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:30:08.549 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:30:08.549 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:30:08.552 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:30:08.603 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:30:08.605 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T04:30:08.673 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:30:08.674 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:30:08.692 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:30:08.692 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:30:08.742 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:30:09.914 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:32:09.965 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:32:10.008 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:32:11.832 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:32:11.850 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:32:11.850 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:32:11.851 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:32:11.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:32:12.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:32:12.101 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:32:12.399 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T04:32:12.430 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:32:12.430 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:32:12.457 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:32:12.458 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:32:12.460 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:32:12.501 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:32:12.571 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:32:12.571 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:32:12.603 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:32:12.642 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:32:12.658 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T04:32:12.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:32:12.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:32:12.726 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:32:12.726 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:32:12.774 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:32:13.827 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:34:13.875 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:34:13.913 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:34:14.979 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:34:14.998 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:34:14.998 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:34:14.998 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:34:14.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:34:15.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:34:15.269 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:34:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:34:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:34:15.633 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:34:15.674 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:34:15.743 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:34:15.743 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:34:15.783 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:34:15.812 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:34:15.880 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:34:15.880 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:34:15.940 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:34:16.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:36:17.039 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:36:17.076 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:36:18.115 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:36:18.140 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:36:18.140 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:36:18.140 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:36:18.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:36:18.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:36:18.410 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:36:18.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:36:18.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:36:18.765 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:36:18.801 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:36:18.872 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:36:18.872 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:36:18.912 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:36:18.943 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:36:19.092 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2021-03-20T04:36:19.236 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2021-03-20T04:36:19.360 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2021-03-20T04:36:19.391 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:36:19.391 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:36:19.564 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:36:20.610 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:38:20.660 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:38:20.697 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:38:21.766 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:38:21.786 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:38:21.786 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:38:21.787 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:38:21.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:38:21.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:38:22.033 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:38:22.381 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:38:22.381 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:38:22.413 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:38:22.451 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:38:22.521 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:38:22.521 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:38:22.580 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:38:22.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:38:22.621 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-20T04:38:22.621 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-20T04:38:22.624 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T04:38:22.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:38:23.821 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:40:23.871 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:40:23.911 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:40:25.048 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:40:25.065 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:40:25.065 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:40:25.065 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:40:25.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:40:25.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:40:25.280 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T04:40:25.317 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:40:26.370 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:42:26.419 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:42:26.456 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:42:27.509 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:42:27.527 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:42:27.527 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:42:27.527 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:42:27.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:42:27.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:42:27.786 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:42:30.822 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T04:42:30.867 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:42:31.912 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:44:31.960 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:44:32.003 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:44:33.026 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:44:33.043 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:44:33.043 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:44:33.043 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:44:33.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:44:33.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:44:33.284 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:44:33.313 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T04:44:34.371 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T04:44:35.473 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T04:44:36.571 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T04:44:37.668 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T04:44:38.765 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T04:44:39.865 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T04:44:40.962 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T04:44:42.060 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T04:44:43.160 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T04:44:43.160 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T04:44:43.196 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:44:44.274 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:46:44.323 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:46:44.380 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:46:45.469 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:46:45.492 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:46:45.493 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:46:45.493 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:46:45.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:46:45.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:46:45.737 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:47:17.768 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T04:47:17.803 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:47:17.804 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:47:17.820 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:47:17.820 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:47:17.822 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:47:17.876 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T04:47:17.877 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:47:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:47:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:47:17.964 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:47:17.965 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:47:17.967 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:47:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:47:18.023 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T04:47:18.084 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:47:18.085 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:47:18.099 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:47:18.099 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:47:18.139 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:47:19.169 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:49:19.218 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:49:19.256 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:49:20.389 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:49:20.407 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:49:20.407 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:49:20.407 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:49:20.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:49:20.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:49:20.662 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:49:20.922 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T04:49:20.953 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:49:20.953 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:49:20.979 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:49:20.979 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:49:20.980 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:49:21.023 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:49:21.036 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T04:49:21.092 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:49:21.092 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:49:21.107 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:49:21.108 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:49:21.109 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:49:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:49:21.162 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T04:49:21.229 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:49:21.230 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:49:21.245 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:49:21.245 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:49:21.284 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:49:22.347 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:51:22.399 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:51:22.486 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:51:23.614 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:51:23.633 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:51:23.633 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:51:23.633 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:51:23.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:51:23.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:51:23.875 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:51:26.914 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T04:51:26.952 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:51:27.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:53:28.042 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:53:28.080 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:53:29.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:53:29.158 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:53:29.158 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:53:29.159 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:53:29.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:53:29.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:53:29.404 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:53:29.690 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T04:53:29.722 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:53:29.722 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:53:29.748 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:53:29.748 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:53:29.750 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:53:29.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:53:29.861 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:53:29.861 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:53:29.897 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:53:29.930 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:53:29.948 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T04:53:29.999 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:53:29.999 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:53:30.015 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:53:30.015 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:53:30.054 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:53:31.076 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:55:31.123 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:55:31.179 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:55:32.272 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:55:32.289 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:55:32.289 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:55:32.289 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:55:32.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:55:32.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:55:32.551 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:55:32.864 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:55:32.864 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:55:32.898 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:55:32.934 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:55:33.003 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:55:33.003 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:55:33.036 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:55:33.070 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:55:33.138 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:55:33.139 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:55:33.197 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:55:34.267 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:57:34.317 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:57:34.360 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:57:35.474 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:57:35.491 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:57:35.491 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:57:35.491 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:57:35.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:57:35.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:57:35.740 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:57:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:57:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:57:36.111 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:57:36.148 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:57:36.217 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:57:36.217 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:57:36.254 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:57:36.285 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:57:36.353 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:57:36.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:57:36.412 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:57:37.467 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:59:37.514 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T04:59:37.553 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:59:38.617 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T04:59:38.634 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T04:59:38.634 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T04:59:38.634 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:59:38.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:59:38.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T04:59:38.879 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T04:59:39.218 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T04:59:39.218 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:59:39.248 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:59:39.288 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T04:59:39.360 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T04:59:39.360 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:59:39.401 INFO:teuthology.misc:Compressing logs... 2021-03-20T04:59:39.429 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T04:59:39.498 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T04:59:39.498 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T04:59:39.570 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T04:59:40.598 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:01:40.656 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:01:40.694 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:01:41.840 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:01:41.877 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:01:41.877 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:01:41.877 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:01:41.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:01:42.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:01:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:01:42.443 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:01:42.443 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:01:42.477 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:01:42.514 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:01:42.585 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:01:42.585 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:01:42.629 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:01:42.657 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:01:42.712 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:01:42.714 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:01:42.751 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:01:43.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:03:43.844 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:03:43.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:03:45.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:03:45.948 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:03:45.948 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:03:45.949 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:03:45.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:03:46.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:03:46.196 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:03:46.550 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:03:46.550 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:03:46.593 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:03:46.627 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:03:46.702 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:03:46.703 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:03:46.767 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:03:46.817 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:03:46.831 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:03:46.832 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:03:46.834 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:03:46.877 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:03:48.391 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:05:48.444 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:05:48.489 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:05:50.249 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:05:50.265 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:05:50.266 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:05:50.266 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:05:50.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:05:50.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:05:50.512 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:05:50.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:05:50.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:05:50.881 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:05:50.919 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:05:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:05:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:05:51.033 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:05:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:05:51.132 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T05:05:51.133 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:05:51.153 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:05:51.155 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:05:51.194 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:05:52.264 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:07:52.311 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:07:52.352 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:07:53.498 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:07:53.517 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:07:53.517 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:07:53.517 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:07:53.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:07:53.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:07:53.777 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:07:54.099 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:07:54.099 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:07:54.130 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:07:54.169 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:07:54.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:07:54.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:07:54.278 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:07:54.310 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:07:54.335 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T05:07:54.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:07:54.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:07:54.336 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:07:54.373 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:07:55.515 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:09:55.568 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:09:55.606 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:09:56.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:09:56.749 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:09:56.749 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:09:56.749 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:09:56.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:09:56.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:09:56.992 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:09:57.326 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:09:57.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:09:57.359 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:09:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:09:57.466 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:09:57.467 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:09:57.508 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:09:57.535 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:09:57.568 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T05:09:57.568 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T05:09:57.568 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T05:09:57.568 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:09:57.568 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T05:09:57.568 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:09:57.569 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:09:57.606 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:09:58.729 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:11:58.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:11:58.815 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:11:59.932 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:11:59.950 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:11:59.951 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:11:59.951 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:11:59.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:12:00.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:12:00.196 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:12:00.529 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:12:00.529 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:12:00.565 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:12:00.599 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:12:00.671 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:12:00.671 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:12:00.713 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:12:00.743 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:12:00.752 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T05:12:00.752 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T05:12:00.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T05:12:00.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:12:00.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T05:12:00.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:12:00.754 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:12:00.795 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:12:01.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:14:01.929 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:14:01.966 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:14:03.155 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:14:03.182 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:14:03.183 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:14:03.183 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:14:03.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:14:03.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:14:03.446 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:14:03.747 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:14:03.747 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:14:03.779 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:14:03.816 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:14:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:14:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:14:03.927 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:14:03.956 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:14:03.966 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T05:14:03.966 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T05:14:03.967 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T05:14:03.982 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:14:03.982 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T05:14:03.982 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:14:03.984 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:14:04.056 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:14:05.199 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:16:05.265 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:16:05.307 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:16:07.364 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:16:07.383 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:16:07.383 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:16:07.383 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:16:07.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:16:07.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:16:07.635 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:16:07.984 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:16:07.985 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:16:08.018 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:16:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:16:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:16:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:16:08.160 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:16:08.193 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:16:08.220 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T05:16:08.220 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T05:16:08.220 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T05:16:08.220 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:16:08.220 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T05:16:08.220 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:16:08.221 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:16:08.259 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:16:09.363 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:18:09.416 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:18:09.456 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:18:10.597 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:18:10.614 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:18:10.614 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:18:10.614 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:18:10.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:18:10.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:18:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:18:11.182 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:18:11.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:18:11.212 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:18:11.253 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:18:11.325 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:18:11.325 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:18:11.371 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:18:11.399 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:18:11.425 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T05:18:11.425 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T05:18:11.425 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T05:18:11.425 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:18:11.426 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T05:18:11.426 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:18:11.426 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:18:11.466 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:18:12.641 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:20:12.705 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:20:12.745 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:20:13.828 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:20:13.845 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:20:13.845 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:20:13.845 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:20:13.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:20:14.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:20:14.108 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:20:14.422 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:20:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:20:14.451 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:20:14.493 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:20:14.563 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:20:14.563 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:20:14.604 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:20:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:20:14.659 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T05:20:14.659 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T05:20:14.659 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T05:20:14.660 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:20:14.660 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T05:20:14.660 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:20:14.662 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:20:14.731 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:20:15.771 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:22:15.821 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:22:15.858 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:22:16.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:22:16.919 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:22:16.919 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:22:16.919 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:22:16.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:22:17.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:22:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:22:17.513 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:22:17.513 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:22:17.543 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:22:17.585 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:22:17.655 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:22:17.655 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:22:17.695 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:22:17.724 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:22:17.752 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T05:22:17.752 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T05:22:17.752 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T05:22:17.752 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:22:17.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T05:22:17.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:22:17.754 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:22:17.791 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:22:18.821 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:24:18.879 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:24:18.918 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:24:20.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:24:20.185 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:24:20.185 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:24:20.186 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:24:20.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:24:20.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:24:20.439 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:24:20.816 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:24:20.817 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:24:20.848 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:24:20.887 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:24:20.958 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:24:20.958 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:24:21.004 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:24:21.042 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:24:21.051 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T05:24:21.052 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T05:24:21.052 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T05:24:21.053 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:24:21.053 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T05:24:21.053 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:24:21.055 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:24:21.155 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:24:22.259 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:26:22.306 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:26:22.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:26:23.364 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:26:23.381 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:26:23.381 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:26:23.382 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:26:23.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:26:23.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:26:23.642 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:26:23.964 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:26:23.964 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:26:23.995 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:26:24.034 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:26:24.104 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:26:24.104 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:26:24.146 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:26:24.173 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:26:24.200 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-20T05:26:24.201 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-20T05:26:24.201 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-20T05:26:24.201 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-20T05:26:24.201 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/a5cf5b80-8938-11eb-909e-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-20T05:26:24.201 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-20T05:26:24.201 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:26:24.239 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:26:25.288 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:28:25.342 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:28:25.469 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:28:26.615 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:28:26.634 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:28:26.634 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:28:26.634 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:28:26.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:28:26.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:28:26.878 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:28:45.250 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T05:28:45.290 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:28:46.394 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:30:46.446 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:30:46.485 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:30:47.486 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:30:47.503 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:30:47.503 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:30:47.503 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:30:47.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:30:47.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:30:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:30:50.790 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T05:30:50.826 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:30:51.870 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:32:51.919 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:32:51.958 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:32:53.056 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:32:53.073 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:32:53.073 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:32:53.073 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:32:53.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:32:53.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:32:53.318 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T05:32:53.358 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:32:54.386 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:34:54.439 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:34:54.478 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:34:55.632 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:34:55.649 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:34:55.650 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:34:55.650 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:34:55.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:34:55.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:34:55.898 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:34:56.099 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:34:56.130 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:34:56.131 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:34:56.183 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:34:56.184 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:34:56.185 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:34:56.212 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:34:56.251 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:34:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:34:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:34:56.316 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:34:56.316 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:34:56.317 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:34:56.348 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:34:56.382 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:34:56.416 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T05:34:56.416 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:34:56.447 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T05:34:56.448 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:34:56.486 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:34:57.506 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:36:57.551 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:36:57.623 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:36:58.650 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:36:58.667 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:36:58.667 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:36:58.667 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:36:58.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:36:58.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:36:58.907 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:37:58.881 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T05:37:58.919 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:37:59.942 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:39:59.992 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:40:00.037 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:40:01.202 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:40:01.219 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:40:01.220 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:40:01.220 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:40:01.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:40:01.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:40:01.491 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:40:01.695 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:40:01.728 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:40:01.728 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:40:01.777 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:40:01.777 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:40:01.778 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:40:01.807 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:40:01.878 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T05:40:01.878 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:40:01.934 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:40:01.963 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:40:01.983 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T05:40:02.032 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T05:40:02.032 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:40:02.056 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T05:40:02.057 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:40:02.101 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:40:03.256 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:42:03.305 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:42:03.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:42:04.482 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:42:04.501 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:42:04.501 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:42:04.501 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:42:04.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:42:04.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:42:04.748 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:42:04.949 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:42:04.981 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:42:04.981 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:42:05.019 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:42:05.019 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:42:05.020 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:42:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:42:05.087 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T05:42:05.089 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:42:05.129 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:42:06.219 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:44:06.271 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:44:06.309 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:44:07.320 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:44:07.337 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:44:07.337 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:44:07.337 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:44:07.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:44:07.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:44:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:44:07.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:44:07.823 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:44:07.861 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:44:07.892 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:44:07.928 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T05:44:07.929 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:44:07.966 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:44:08.956 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:46:09.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:46:09.058 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:46:10.098 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:46:10.115 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:46:10.115 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:46:10.115 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:46:10.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:46:10.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:46:10.380 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:46:10.607 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:46:10.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:46:10.647 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:46:10.676 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:46:10.713 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T05:46:10.714 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:46:10.753 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:46:11.797 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:48:11.853 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:48:11.891 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:48:13.028 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:48:13.046 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:48:13.046 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:48:13.046 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:48:13.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:48:13.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:48:13.295 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:48:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:48:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:48:13.573 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:48:13.610 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:48:13.638 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T05:48:13.639 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:48:13.683 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:48:14.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:50:14.868 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:50:14.912 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:50:17.090 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:50:17.106 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:50:17.107 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:50:17.107 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:50:17.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:50:17.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:50:17.354 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:50:17.603 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:50:17.603 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:50:17.631 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:50:17.674 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:50:17.697 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T05:50:17.699 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:50:17.739 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:50:18.957 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:52:19.005 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:52:19.052 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:52:20.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:52:20.278 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:52:20.278 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:52:20.278 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:52:20.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:52:20.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:52:20.537 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:52:20.761 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:52:20.761 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:52:20.800 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:52:20.830 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:52:20.865 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T05:52:20.867 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:52:20.904 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:52:22.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:54:22.059 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:54:22.098 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:54:23.187 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:54:23.205 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:54:23.205 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:54:23.205 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:54:23.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:54:23.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:54:23.449 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:54:23.690 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:54:23.690 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:54:23.730 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:54:23.758 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:54:23.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T05:54:23.798 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:54:23.878 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:54:24.923 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:56:24.971 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:56:25.008 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:56:26.086 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:56:26.103 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:56:26.103 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:56:26.103 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:56:26.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:56:26.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:56:26.347 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:56:26.603 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:56:26.603 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:56:26.631 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:56:26.673 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:56:26.698 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T05:56:26.699 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:56:26.735 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:56:27.806 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:58:27.867 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T05:58:27.907 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:58:29.038 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T05:58:29.055 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T05:58:29.055 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T05:58:29.055 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:58:29.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:58:29.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T05:58:29.311 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T05:58:29.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T05:58:29.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T05:58:29.578 INFO:teuthology.misc:Compressing logs... 2021-03-20T05:58:29.606 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T05:58:29.645 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T05:58:29.647 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T05:58:29.684 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T05:58:30.732 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:00:30.780 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:00:30.822 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:00:31.967 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:00:32.001 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:00:32.001 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:00:32.001 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:00:32.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:00:32.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:00:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:00:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:00:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:00:32.521 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:00:32.553 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:00:32.587 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:00:32.589 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:00:32.682 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:00:33.804 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:02:33.855 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:02:33.893 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:02:34.923 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:02:34.940 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:02:34.940 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:02:34.941 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:02:34.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:02:35.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:02:35.188 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:02:35.440 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:02:35.440 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:02:35.469 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:02:35.509 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:02:35.535 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:02:35.536 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:02:35.576 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:02:36.641 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:04:36.689 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:04:36.729 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:04:38.228 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:04:38.245 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:04:38.245 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:04:38.245 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:04:38.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:04:38.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:04:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:04:38.733 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:04:38.734 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:04:38.775 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:04:38.804 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:04:38.842 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:04:38.843 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:04:38.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:04:39.975 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:06:40.029 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:06:40.074 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:06:41.187 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:06:41.204 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:06:41.204 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:06:41.204 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:06:41.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:06:41.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:06:41.449 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:06:41.691 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:06:41.691 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:06:41.730 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:06:41.761 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:06:41.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:06:41.798 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:06:41.837 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:06:42.937 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:08:42.989 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:08:43.026 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:08:44.279 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:08:44.298 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:08:44.298 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:08:44.298 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:08:44.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:08:44.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:08:44.540 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:08:44.794 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:08:44.795 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:08:44.825 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:08:44.865 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:08:44.891 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:08:44.892 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:08:44.931 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:08:45.986 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:10:46.037 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:10:47.195 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:10:48.266 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:10:48.283 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:10:48.283 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:10:48.283 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:10:48.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:10:48.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:10:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:10:48.773 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:10:48.773 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:10:48.814 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:10:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:10:48.881 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:10:48.882 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:10:48.920 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:10:50.033 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:12:50.081 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:12:50.119 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:12:51.916 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:12:51.936 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:12:51.936 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:12:51.936 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:12:51.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:12:52.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:12:52.249 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:12:52.594 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:12:52.594 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:12:52.645 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:12:52.676 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:12:52.730 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:12:52.732 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:12:52.778 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:12:54.252 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:14:54.302 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:14:54.340 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:14:56.175 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:14:56.192 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:14:56.192 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:14:56.192 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:14:56.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:14:56.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:14:56.436 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:14:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:14:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:14:56.716 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:14:56.757 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:14:56.781 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:14:56.783 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:14:56.821 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:14:57.909 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:16:57.955 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:16:57.997 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:16:59.147 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:16:59.165 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:16:59.165 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:16:59.165 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:16:59.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:16:59.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:16:59.424 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:16:59.648 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:16:59.648 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:16:59.690 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:16:59.716 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:16:59.757 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:16:59.758 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:16:59.797 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:17:00.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:19:00.956 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:19:01.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:19:02.171 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:19:02.190 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:19:02.191 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:19:02.191 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:19:02.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:19:02.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:19:02.439 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:19:02.844 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:19:02.845 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:19:02.883 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:19:02.914 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:19:02.949 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:19:02.951 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:19:02.998 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:19:04.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:21:04.242 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:21:04.305 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:21:05.449 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:21:05.467 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:21:05.467 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:21:05.467 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:21:05.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:21:05.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:21:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:21:05.971 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:21:05.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:21:06.004 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:21:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:21:06.071 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:21:06.072 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:21:06.110 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:21:07.196 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:23:07.251 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:23:07.290 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:23:09.195 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:23:09.213 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:23:09.213 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:23:09.213 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:23:09.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:23:09.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:23:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:23:09.806 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:23:09.807 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:23:09.847 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:23:09.877 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:23:09.915 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:23:09.916 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:23:09.955 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:23:11.330 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:25:11.391 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:25:11.447 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:25:13.817 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:25:13.854 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:25:13.855 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:25:13.855 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:25:13.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:25:14.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:25:14.113 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:25:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:25:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:25:14.397 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:25:14.430 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:25:14.465 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:25:14.465 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:25:14.517 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:25:16.977 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:27:17.029 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:27:17.143 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:27:19.636 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:27:19.655 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:27:19.655 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:27:19.655 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:27:19.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:27:19.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:27:19.907 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:27:20.166 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:27:20.167 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:27:20.199 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:27:20.238 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:27:20.266 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:27:20.268 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:27:20.310 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:27:22.647 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:29:22.698 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:29:22.751 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:29:24.673 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:29:24.703 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:29:24.703 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:29:24.704 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:29:24.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:29:24.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:29:24.967 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:29:25.200 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:29:25.201 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:29:25.244 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:29:25.271 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:29:25.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:29:25.312 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:29:25.356 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:29:27.761 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:31:27.814 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:31:27.862 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:31:29.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:31:29.956 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:31:29.957 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:31:29.957 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:31:29.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:31:30.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:31:30.211 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:31:30.457 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:31:30.458 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:31:30.498 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:31:30.529 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:31:30.567 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:31:30.567 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:31:30.611 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:31:32.945 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:33:33.002 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:33:33.046 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:33:35.299 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:33:35.316 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:33:35.316 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:33:35.316 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:33:35.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:33:35.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:33:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:33:35.823 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:33:35.824 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:33:35.866 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:33:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:33:35.932 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:33:35.935 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:33:35.983 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:33:37.586 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:35:37.655 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:35:37.698 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:35:39.862 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:35:39.956 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:35:39.956 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:35:39.957 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:35:39.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:35:40.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:35:40.218 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:35:40.443 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:35:40.444 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:35:40.485 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:35:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:35:40.552 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:35:40.554 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:35:40.604 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:35:41.753 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:37:41.807 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:37:41.853 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:37:43.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:37:43.994 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:37:43.995 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:37:43.995 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:37:43.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:37:44.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:37:44.243 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:37:44.489 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:37:44.490 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:37:44.529 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:37:44.561 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:37:44.596 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:37:44.599 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:37:44.647 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:37:46.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:39:46.742 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:39:46.787 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:39:48.983 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:39:49.003 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:39:49.003 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:39:49.003 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:39:49.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:39:49.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:39:49.254 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:39:49.519 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:39:49.519 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:39:49.552 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:39:49.589 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:39:49.619 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:39:49.620 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:39:49.661 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:39:51.849 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:41:51.915 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:41:51.960 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:41:54.256 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:41:54.284 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:41:54.284 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:41:54.285 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:41:54.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:41:54.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:41:54.549 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:41:54.776 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:41:54.776 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:41:54.818 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:41:54.846 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:41:54.885 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:41:54.887 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:41:54.947 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:41:56.872 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:43:56.927 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:43:56.968 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:43:58.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:43:58.251 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:43:58.252 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:43:58.252 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:43:58.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:43:58.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:43:58.505 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:43:58.758 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:43:58.758 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:43:58.799 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:43:58.831 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:43:58.867 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:43:58.868 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:43:58.914 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:44:00.234 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:46:00.288 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:46:00.330 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:46:01.444 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:46:01.481 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:46:01.481 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:46:01.482 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:46:01.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:46:01.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:46:01.731 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:46:01.987 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:46:01.987 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:46:02.017 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:46:02.058 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:46:02.083 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:46:02.085 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:46:02.126 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:46:03.259 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:48:03.311 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:48:03.439 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:48:04.582 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:48:04.606 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:48:04.607 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:48:04.607 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:48:04.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:48:04.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:48:04.874 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:48:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:48:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:48:05.139 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:48:05.167 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:48:05.207 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:48:05.208 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:48:05.253 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:48:06.377 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:50:06.436 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:50:06.484 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:50:07.620 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:50:07.637 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:50:07.638 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:50:07.638 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:50:07.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:50:07.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:50:07.882 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:50:08.126 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:50:08.126 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:50:08.166 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:50:08.195 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:50:08.232 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:50:08.234 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:50:08.275 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:50:09.425 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:52:09.476 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:52:09.516 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:52:10.618 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:52:10.636 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:52:10.636 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:52:10.636 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:52:10.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:52:10.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:52:10.885 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:52:11.146 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:52:11.147 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:52:11.179 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:52:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:52:11.249 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:52:11.249 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:52:11.303 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:52:12.347 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:54:12.398 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:54:12.437 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:54:13.564 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:54:13.584 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:54:13.584 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:54:13.584 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:54:13.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:54:13.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:54:13.844 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:54:14.064 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:54:14.064 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:54:14.105 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:54:14.133 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:54:14.172 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:54:14.173 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:54:14.226 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:54:15.342 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:56:15.397 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:56:15.436 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:56:16.473 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:56:16.491 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:56:16.491 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:56:16.491 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:56:16.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:56:16.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:56:16.734 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:56:16.977 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:56:16.978 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:56:17.018 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:56:17.047 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:56:17.084 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:56:17.086 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:56:17.152 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:56:18.221 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:58:18.270 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T06:58:18.317 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:58:19.499 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T06:58:19.522 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T06:58:19.523 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T06:58:19.523 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:58:19.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:58:19.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T06:58:19.768 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T06:58:20.028 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T06:58:20.028 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T06:58:20.061 INFO:teuthology.misc:Compressing logs... 2021-03-20T06:58:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T06:58:20.127 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T06:58:20.128 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T06:58:20.176 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T06:58:21.312 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:00:21.367 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:00:21.412 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:00:22.589 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:00:22.615 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:00:22.615 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:00:22.615 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:00:22.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:00:22.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:00:22.876 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:00:23.101 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:00:23.102 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:00:23.143 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:00:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:00:23.210 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:00:23.211 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:00:23.249 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:00:24.314 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:02:24.369 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:02:24.410 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:02:25.499 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:02:25.516 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:02:25.516 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:02:25.516 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:02:25.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:02:25.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:02:25.761 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:02:26.004 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:02:26.004 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:02:26.043 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:02:26.075 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:02:26.109 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:02:26.110 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:02:26.148 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:02:27.220 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:04:27.269 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:04:27.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:04:28.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:04:28.384 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:04:28.384 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:04:28.384 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:04:28.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:04:28.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:04:28.626 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:04:28.882 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:04:28.882 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:04:28.912 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:04:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:04:28.978 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:04:28.979 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:04:29.016 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:04:30.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:06:30.100 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:06:30.139 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:06:31.202 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:06:31.223 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:06:31.223 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:06:31.223 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:06:31.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:06:31.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:06:31.485 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:06:31.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:06:31.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:06:31.752 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:06:31.780 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:06:31.818 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:06:31.820 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:06:31.857 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:06:32.868 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:08:32.924 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:08:32.965 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:08:34.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:08:34.875 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:08:34.875 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:08:34.875 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:08:34.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:08:35.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:08:35.133 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:08:35.370 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:08:35.370 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:08:35.411 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:08:35.441 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:08:35.476 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:08:35.478 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:08:35.519 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:08:36.838 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:10:36.889 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:10:37.000 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:10:39.100 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:10:39.117 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:10:39.117 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:10:39.117 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:10:39.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:10:39.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:10:39.365 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:10:39.627 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:10:39.627 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:10:39.657 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:10:39.699 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:10:39.726 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:10:39.726 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:10:39.768 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:10:41.025 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:12:41.076 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:12:41.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:12:42.955 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:12:42.973 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:12:42.973 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:12:42.973 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:12:42.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:12:43.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:12:43.232 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:12:43.460 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:12:43.460 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:12:43.501 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:12:43.528 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:12:43.567 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:12:43.568 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:12:43.609 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:12:44.697 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:14:44.748 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:14:44.800 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:14:46.463 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:14:46.482 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:14:46.482 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:14:46.482 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:14:46.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:14:46.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:14:46.734 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:14:46.975 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:14:46.975 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:14:47.014 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:14:47.045 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:14:47.080 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:14:47.082 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:14:47.120 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:14:48.187 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:16:48.237 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:16:48.279 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:16:49.407 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:16:49.426 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:16:49.426 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:16:49.426 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:16:49.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:16:49.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:16:49.673 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:16:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:16:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:16:49.954 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:16:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:16:50.020 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:16:50.022 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:16:50.061 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:16:51.213 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:18:51.266 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:18:51.335 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:18:53.203 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:18:53.222 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:18:53.222 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:18:53.222 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:18:53.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:18:53.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:18:53.491 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:18:53.721 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:18:53.721 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:18:53.761 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:18:53.791 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:18:53.829 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:18:53.830 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:18:53.884 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:18:55.679 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:20:55.732 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:20:55.781 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:20:57.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:20:57.922 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:20:57.922 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:20:57.922 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:20:57.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:20:58.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:20:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:20:58.415 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:20:58.416 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:20:58.455 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:20:58.485 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:20:58.521 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:20:58.522 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:20:58.570 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:20:59.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:22:59.907 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:22:59.952 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:23:01.991 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:23:02.015 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:23:02.016 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:23:02.016 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:23:02.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:23:02.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:23:02.268 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:23:02.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:23:02.521 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:23:02.551 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:23:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:23:02.617 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:23:02.619 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:23:02.659 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:23:04.548 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:25:04.597 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:25:04.640 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:25:06.720 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:25:06.738 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:25:06.738 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:25:06.738 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:25:06.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:25:06.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:25:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:25:07.229 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:25:07.230 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:25:07.268 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:25:07.300 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:25:07.337 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:25:07.338 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:25:07.381 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:25:08.999 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:27:09.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:27:09.103 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:27:10.975 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:27:10.998 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:27:10.999 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:27:10.999 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:27:10.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:27:11.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:27:11.252 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:27:11.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:27:11.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:27:11.532 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:27:11.560 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:27:11.600 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:27:11.601 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:27:11.639 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:27:12.753 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:29:12.806 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:29:12.851 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:29:14.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:29:14.875 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:29:14.875 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:29:14.875 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:29:14.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:29:15.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:29:15.130 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:29:15.389 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:29:15.389 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:29:15.421 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:29:15.461 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:29:15.487 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:29:15.489 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:29:15.527 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:29:17.650 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:31:17.704 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:31:17.746 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:31:19.684 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:31:19.702 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:31:19.702 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:31:19.702 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:31:19.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:31:19.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:31:19.962 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:31:20.189 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:31:20.190 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:31:20.231 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:31:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:31:20.299 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:31:20.300 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:31:20.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:31:21.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:33:21.826 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:33:21.906 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:33:23.274 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:33:23.295 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:33:23.296 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:33:23.296 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:33:23.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:33:23.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:33:23.547 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:33:23.786 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:33:23.786 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:33:23.826 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:33:23.855 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:33:23.892 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:33:23.893 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:33:23.931 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:33:25.774 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:35:25.827 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:35:25.868 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:35:26.940 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:35:26.962 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:35:26.962 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:35:26.963 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:35:26.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:35:27.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:35:27.207 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:35:27.469 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:35:27.469 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:35:27.503 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:35:27.541 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:35:27.570 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:35:27.572 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:35:27.611 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:35:28.633 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:37:28.691 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:37:28.730 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:37:30.608 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:37:30.625 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:37:30.625 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:37:30.626 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:37:30.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:37:30.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:37:30.885 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:37:31.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:37:31.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:37:31.149 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:37:31.180 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:37:31.216 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:37:31.217 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:37:31.257 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:37:32.385 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:39:32.496 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:39:32.535 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:39:34.470 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:39:34.490 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:39:34.490 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:39:34.490 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:39:34.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:39:34.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:39:34.737 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:39:34.978 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:39:34.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:39:35.017 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:39:35.049 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:39:35.084 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:39:35.085 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:39:35.123 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:39:37.051 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:41:37.105 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:41:37.142 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:41:39.094 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:41:39.114 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:41:39.114 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:41:39.114 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:41:39.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:41:39.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:41:39.431 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:41:39.686 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:41:39.686 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:41:39.715 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:41:39.757 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:41:39.782 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:41:39.783 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:41:39.823 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:41:41.797 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:43:41.862 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:43:41.907 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:43:43.114 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:43:43.146 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:43:43.146 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:43:43.146 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:43:43.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:43:43.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:43:43.412 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:43:43.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:43:43.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:43:43.676 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:43:43.705 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:43:43.743 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:43:43.744 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:43:43.834 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:43:44.949 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:45:45.011 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:45:45.073 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:45:46.216 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:45:46.237 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:45:46.237 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:45:46.238 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:45:46.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:45:46.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:45:46.482 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:45:46.724 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:45:46.724 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:45:46.768 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:45:46.796 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:45:46.835 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:45:46.836 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:45:46.876 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:45:47.946 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:47:48.001 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:47:48.048 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:47:49.063 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:47:49.080 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:47:49.080 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:47:49.081 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:47:49.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:47:49.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:47:49.331 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:47:49.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:47:49.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:47:49.619 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:47:49.658 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:47:49.686 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:47:49.687 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:47:49.725 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:47:50.768 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:49:50.819 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:49:50.861 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:49:51.913 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:49:51.930 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:49:51.931 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:49:51.931 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:49:51.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:49:52.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:49:52.191 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:49:52.414 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:49:52.415 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:49:52.456 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:49:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:49:52.523 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:49:52.525 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:49:52.564 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:49:53.585 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:51:53.634 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:51:53.672 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:51:54.731 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:51:54.748 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:51:54.748 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:51:54.749 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:51:54.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:51:54.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:51:54.990 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:51:55.227 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:51:55.227 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:51:55.268 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:51:55.295 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:51:55.335 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:51:55.337 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:51:55.374 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:51:56.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:53:56.572 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:53:56.613 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:53:57.663 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:53:57.681 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:53:57.681 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:53:57.681 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:53:57.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:53:57.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:53:57.923 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:53:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:53:58.180 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:53:58.211 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:53:58.247 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:53:58.277 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:53:58.279 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:53:58.320 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:53:59.380 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:55:59.431 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:55:59.470 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:56:00.656 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:56:00.675 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:56:00.675 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:56:00.675 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:56:00.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:56:00.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:56:00.935 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:56:01.166 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:56:01.167 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:56:01.206 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:56:01.234 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:56:01.273 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:56:01.275 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:56:01.330 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:56:02.424 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:58:02.480 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T07:58:02.519 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:58:03.569 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T07:58:03.587 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T07:58:03.588 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T07:58:03.588 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:58:03.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:58:03.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T07:58:03.832 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T07:58:04.071 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T07:58:04.071 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T07:58:04.110 INFO:teuthology.misc:Compressing logs... 2021-03-20T07:58:04.142 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T07:58:04.177 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T07:58:04.179 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T07:58:04.221 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T07:58:05.303 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:00:05.352 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:00:05.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:00:06.466 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:00:06.484 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:00:06.484 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:00:06.485 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:00:06.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:00:06.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:00:06.738 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:00:06.998 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:00:06.998 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:00:07.028 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:00:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:00:07.094 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:00:07.101 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:00:07.169 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:00:08.287 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:02:08.419 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:02:08.461 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:02:09.673 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:02:09.692 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:02:09.692 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:02:09.693 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:02:09.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:02:09.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:02:09.956 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:02:10.182 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:02:10.182 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:02:10.222 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:02:10.251 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:02:10.290 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:02:10.291 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:02:10.329 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:02:11.447 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:04:11.496 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:04:11.536 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:04:12.604 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:04:12.622 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:04:12.622 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:04:12.623 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:04:12.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:04:12.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:04:12.869 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:04:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:04:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:04:13.145 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:04:13.175 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:04:13.211 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:04:13.212 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:04:13.250 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:04:14.309 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:06:14.361 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:06:14.403 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:06:15.451 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:06:15.470 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:06:15.470 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:06:15.470 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:06:15.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:06:15.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:06:15.714 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:06:15.973 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:06:15.973 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:06:16.002 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:06:16.042 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:06:16.068 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:06:16.069 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:06:16.106 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:06:17.192 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:08:17.243 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:08:17.282 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:08:18.425 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:08:18.444 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:08:18.444 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:08:18.444 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:08:18.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:08:18.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:08:18.701 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:08:18.925 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:08:18.926 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:08:18.968 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:08:18.994 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:08:19.034 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:08:19.036 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:08:19.075 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:08:20.096 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:10:20.146 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:10:20.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:10:21.233 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:10:21.250 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:10:21.250 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:10:21.250 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:10:21.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:10:21.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:10:21.493 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:10:21.732 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:10:21.732 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:10:21.773 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:10:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:10:21.839 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:10:21.841 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:10:21.879 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:10:22.897 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:12:22.946 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:12:22.985 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:12:24.019 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:12:24.037 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:12:24.037 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:12:24.037 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:12:24.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:12:24.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:12:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:12:24.536 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:12:24.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:12:24.566 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:12:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:12:24.632 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:12:24.634 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:12:24.670 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:12:25.711 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:14:25.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:14:25.818 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:14:26.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:14:26.880 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:14:26.880 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:14:26.881 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:14:26.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:14:27.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:14:27.141 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:14:27.363 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:14:27.363 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:14:27.404 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:14:27.433 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:14:27.470 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:14:27.472 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:14:27.512 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:14:28.549 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:16:28.604 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:16:28.645 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:16:29.770 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:16:29.787 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:16:29.787 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:16:29.787 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:16:29.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:16:30.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:16:30.029 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:16:30.265 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:16:30.265 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:16:30.307 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:16:30.334 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:16:30.374 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:16:30.375 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:16:30.413 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:16:31.453 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:18:31.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:18:31.589 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:18:32.742 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:18:32.759 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:18:32.759 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:18:32.759 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:18:32.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:18:32.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:18:33.004 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:18:33.256 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:18:33.257 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:18:33.285 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:18:33.325 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:18:33.352 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:18:33.353 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:18:33.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:18:34.523 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:20:34.588 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:20:34.626 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:20:35.727 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:20:35.745 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:20:35.746 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:20:35.746 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:20:35.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:20:35.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:20:36.010 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:20:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:20:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:20:36.276 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:20:36.306 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:20:36.343 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:20:36.344 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:20:36.383 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:20:37.485 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:22:37.541 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:22:37.581 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:22:38.657 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:22:38.674 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:22:38.674 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:22:38.674 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:22:38.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:22:38.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:22:38.919 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:22:39.163 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:22:39.163 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:22:39.201 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:22:39.232 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:22:39.267 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:22:39.268 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:22:39.307 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:22:40.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:24:40.409 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:24:40.464 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:24:41.594 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:24:41.611 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:24:41.611 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:24:41.611 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:24:41.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:24:41.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:24:41.857 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:24:42.114 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:24:42.114 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:24:42.143 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:24:42.183 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:24:42.209 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:24:42.210 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:24:42.254 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:24:43.363 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:26:43.411 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:26:43.448 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:26:44.515 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:26:44.536 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:26:44.536 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:26:44.536 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:26:44.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:26:44.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:26:44.795 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:26:45.023 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:26:45.024 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:26:45.065 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:26:45.092 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:26:45.132 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:26:45.134 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:26:45.175 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:26:46.236 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:28:46.288 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:28:46.332 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:28:47.556 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:28:47.573 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:28:47.573 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:28:47.573 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:28:47.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:28:47.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:28:47.820 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:28:48.069 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:28:48.069 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:28:48.109 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:28:48.140 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:28:48.176 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:28:48.178 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:28:48.219 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:28:49.303 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:30:49.352 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:30:49.391 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:30:51.226 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:30:51.242 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:30:51.243 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:30:51.243 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:30:51.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:30:51.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:30:51.488 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:30:51.745 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:30:51.745 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:30:51.775 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:30:51.814 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:30:51.841 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:30:51.843 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:30:51.891 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:30:53.220 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:32:53.269 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:32:53.307 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:32:55.098 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:32:55.115 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:32:55.115 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:32:55.115 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:32:55.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:32:55.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:32:55.379 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:32:55.609 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:32:55.609 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:32:55.649 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:32:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:32:55.715 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:32:55.717 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:32:55.761 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:32:56.874 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:34:56.925 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:34:56.963 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:34:58.066 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:34:58.101 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:34:58.102 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:34:58.102 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:34:58.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:34:58.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:34:58.343 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:34:58.585 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:34:58.585 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:34:58.625 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:34:58.655 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:34:58.692 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:34:58.693 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:34:58.731 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:34:59.946 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:37:00.232 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:37:00.275 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:37:02.496 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:37:02.524 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:37:02.524 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:37:02.524 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:37:02.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:37:02.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:37:02.779 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:37:03.027 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:37:03.027 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:37:03.059 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:37:03.097 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:37:03.125 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:37:03.128 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:37:03.168 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:37:04.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:39:04.886 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:39:04.926 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:39:06.846 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:39:06.864 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:39:06.864 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:39:06.864 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:39:06.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:39:07.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:39:07.125 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:39:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:39:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:39:07.395 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:39:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:39:07.462 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:39:07.463 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:39:07.504 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:39:08.586 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:41:08.637 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:41:08.678 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:41:10.098 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:41:10.122 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:41:10.122 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:41:10.122 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:41:10.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:41:10.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:41:10.366 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:41:10.614 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:41:10.614 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:41:10.655 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:41:10.683 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:41:10.722 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:41:10.723 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:41:10.765 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:41:11.857 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:43:11.908 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:43:11.949 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:43:13.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:43:13.030 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:43:13.030 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:43:13.030 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:43:13.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:43:13.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:43:13.276 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:43:13.525 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:43:13.525 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:43:13.555 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:43:13.592 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:43:13.622 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:43:13.623 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:43:13.669 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:43:14.728 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:45:14.776 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:45:14.819 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:45:16.829 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:45:16.847 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:45:16.847 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:45:16.847 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:45:16.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:45:17.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:45:17.113 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:45:17.341 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:45:17.342 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:45:17.387 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:45:17.429 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:45:17.487 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:45:17.488 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:45:17.528 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:45:18.970 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:47:19.021 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:47:19.059 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:47:20.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:47:20.854 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:47:20.854 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:47:20.854 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:47:20.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:47:21.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:47:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:47:21.333 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:47:21.333 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:47:21.373 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:47:21.402 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:47:21.440 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:47:21.441 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:47:21.479 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:47:22.857 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:49:22.911 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:49:23.066 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:49:24.161 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:49:24.178 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:49:24.178 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:49:24.178 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:49:24.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:49:24.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:49:24.424 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:49:24.680 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:49:24.680 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:49:24.710 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:49:24.752 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:49:24.777 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:49:24.779 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:49:24.815 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:49:25.955 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:51:26.012 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:51:26.144 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:51:28.115 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:51:28.132 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:51:28.132 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:51:28.132 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:51:28.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:51:28.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:51:28.395 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:51:28.624 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:51:28.624 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:51:28.663 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:51:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:51:28.730 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:51:28.732 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:51:28.832 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:51:29.973 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:53:30.023 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:53:30.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:53:31.608 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:53:31.627 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:53:31.628 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:53:31.628 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:53:31.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:53:31.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:53:31.871 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:53:32.110 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:53:32.110 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:53:32.148 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:53:32.178 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:53:32.216 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:53:32.218 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:53:32.255 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:53:33.363 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:55:33.414 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:55:33.482 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:55:35.709 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:55:35.730 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:55:35.730 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:55:35.730 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:55:35.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:55:35.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:55:35.980 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:55:36.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:55:36.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:55:36.271 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:55:36.311 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:55:36.338 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:55:36.340 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:55:36.407 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:55:38.595 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:57:38.715 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:57:39.137 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:57:41.099 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:57:41.116 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:57:41.116 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:57:41.117 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:57:41.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:57:41.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:57:41.375 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:57:41.603 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:57:41.603 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:57:41.643 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:57:41.669 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:57:41.711 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:57:41.712 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:57:41.751 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:57:42.818 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:59:42.871 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T08:59:42.909 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:59:45.083 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T08:59:45.115 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T08:59:45.116 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T08:59:45.116 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:59:45.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:59:45.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T08:59:45.366 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T08:59:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T08:59:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T08:59:45.645 INFO:teuthology.misc:Compressing logs... 2021-03-20T08:59:45.674 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T08:59:45.711 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T08:59:45.713 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T08:59:45.769 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T08:59:47.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:01:47.120 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:01:47.196 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:01:49.441 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:01:49.473 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:01:49.474 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:01:49.474 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:01:49.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:01:49.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:01:49.726 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:01:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:01:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:01:50.023 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:01:50.057 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:01:50.090 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:01:50.092 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:01:50.145 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:01:52.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:03:52.600 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:03:52.679 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:03:54.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:03:54.837 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:03:54.837 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:03:54.837 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:03:54.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:03:55.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:03:55.100 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:03:55.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:03:55.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:03:55.368 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:03:55.399 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:03:55.437 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:03:55.438 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:03:55.481 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:03:57.477 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:05:57.529 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:05:57.568 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:05:58.650 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:05:58.670 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:05:58.671 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:05:58.671 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:05:58.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:05:58.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:05:58.921 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:05:59.161 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:05:59.161 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:05:59.200 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:05:59.231 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:05:59.266 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:05:59.268 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:05:59.306 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:06:00.356 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:08:00.406 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:08:00.448 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:08:01.482 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:08:01.501 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:08:01.502 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:08:01.502 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:08:01.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:08:01.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:08:01.745 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:08:02.002 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:08:02.003 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:08:02.032 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:08:02.073 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:08:02.099 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:08:02.101 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:08:02.138 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:08:03.171 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:10:03.220 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:10:03.259 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:10:04.959 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:10:04.976 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:10:04.976 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:10:04.977 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:10:04.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:10:05.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:10:05.237 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:10:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:10:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:10:05.507 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:10:05.533 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:10:05.573 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:10:05.575 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:10:05.616 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:10:07.561 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:12:07.615 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:12:07.657 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:12:09.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:12:09.064 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:12:09.066 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:12:09.066 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:12:09.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:12:09.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:12:09.311 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:12:09.549 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:12:09.549 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:12:09.588 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:12:09.618 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:12:09.653 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:12:09.654 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:12:09.693 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:12:10.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:14:10.901 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:14:10.943 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:14:12.134 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:14:12.153 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:14:12.153 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:14:12.153 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:14:12.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:14:12.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:14:12.400 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:14:12.699 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:14:12.699 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:14:12.729 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:14:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:14:12.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:14:12.797 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:14:12.836 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:14:13.932 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:16:13.987 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:16:14.026 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:16:15.165 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:16:15.184 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:16:15.184 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:16:15.184 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:16:15.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:16:15.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:16:15.444 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:16:15.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:16:15.669 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:16:15.710 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:16:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:16:15.778 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:16:15.780 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:16:15.824 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:16:17.178 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:18:17.231 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:18:17.275 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:18:19.229 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:18:19.254 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:18:19.255 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:18:19.255 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:18:19.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:18:19.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:18:19.499 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:18:19.743 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:18:19.743 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:18:19.782 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:18:19.814 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:18:19.849 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:18:19.851 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:18:19.903 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:18:21.286 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:20:21.362 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:20:21.435 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:20:22.721 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:20:22.740 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:20:22.740 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:20:22.740 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:20:22.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:20:22.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:20:22.986 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:20:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:20:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:20:23.268 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:20:23.306 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:20:23.335 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:20:23.336 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:20:23.376 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:20:24.446 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:22:24.496 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:22:24.538 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:22:25.606 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:22:25.624 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:22:25.624 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:22:25.624 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:22:25.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:22:25.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:22:25.888 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:22:26.114 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:22:26.115 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:22:26.155 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:22:26.183 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:22:26.221 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:22:26.223 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:22:26.260 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:22:27.356 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:24:27.405 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:24:27.510 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:24:28.569 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:24:28.586 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:24:28.586 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:24:28.586 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:24:28.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:24:28.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:24:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:24:29.080 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:24:29.080 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:24:29.118 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:24:29.150 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:24:29.185 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:24:29.187 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:24:29.225 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:24:30.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:26:30.313 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:26:30.351 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:26:31.464 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:26:31.483 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:26:31.483 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:26:31.483 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:26:31.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:26:31.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:26:31.727 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:26:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:26:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:26:32.028 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:26:32.064 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:26:32.095 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:26:32.097 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:26:32.193 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:26:33.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:28:33.324 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:28:33.361 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:28:34.472 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:28:34.489 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:28:34.489 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:28:34.489 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:28:34.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:28:34.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:28:34.751 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:28:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:28:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:28:35.016 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:28:35.046 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:28:35.082 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:28:35.084 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:28:35.121 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:28:36.165 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:30:36.218 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:30:36.257 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:30:37.461 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:30:37.478 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:30:37.478 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:30:37.479 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:30:37.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:30:37.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:30:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:30:37.970 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:30:37.971 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:30:38.010 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:30:38.039 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:30:38.076 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:30:38.078 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:30:38.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:30:39.226 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:32:39.274 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:32:39.312 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:32:40.381 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:32:40.398 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:32:40.398 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:32:40.398 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:32:40.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:32:40.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:32:40.647 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:32:40.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:32:40.904 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:32:40.935 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:32:40.973 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:32:41.002 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:32:41.003 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:32:41.041 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:32:42.144 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:34:42.192 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:34:42.236 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:34:44.375 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:34:44.394 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:34:44.394 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:34:44.394 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:34:44.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:34:44.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:34:44.659 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:34:44.887 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:34:44.887 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:34:44.927 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:34:44.957 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:34:44.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:34:44.996 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:34:45.040 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:34:46.922 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:36:46.975 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:36:47.023 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:36:49.350 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:36:49.367 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:36:49.367 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:36:49.368 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:36:49.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:36:49.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:36:49.614 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:36:49.861 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:36:49.862 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:36:49.898 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:36:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:36:49.965 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:36:49.966 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:36:50.005 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:36:52.010 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:38:52.062 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:38:52.104 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:38:54.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:38:54.166 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:38:54.167 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:38:54.167 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:38:54.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:38:54.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:38:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:38:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:38:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:38:54.691 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:38:54.729 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:38:54.758 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:38:54.759 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:38:54.800 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:38:56.265 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:40:56.317 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:40:56.357 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:40:58.284 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:40:58.302 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:40:58.303 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:40:58.303 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:40:58.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:40:58.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:40:58.564 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:40:58.832 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:40:58.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:40:58.874 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:40:58.901 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:40:58.942 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:40:58.943 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:40:58.982 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:41:00.099 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:43:00.148 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:43:00.186 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:43:01.252 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:43:01.269 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:43:01.269 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:43:01.269 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:43:01.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:43:01.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:43:01.515 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:43:01.755 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:43:01.755 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:43:01.796 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:43:01.823 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:43:01.863 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:43:01.865 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:43:01.902 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:43:02.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:45:03.028 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:45:03.068 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:45:04.199 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:45:04.216 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:45:04.217 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:45:04.217 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:45:04.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:45:04.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:45:04.456 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:45:04.716 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:45:04.717 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:45:04.746 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:45:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:45:04.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:45:04.813 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:45:04.852 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:45:05.953 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:47:05.999 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:47:06.040 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:47:07.673 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:47:07.690 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:47:07.690 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:47:07.690 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:47:07.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:47:07.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:47:07.955 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:47:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:47:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:47:08.219 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:47:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:47:08.287 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:47:08.289 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:47:08.328 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:47:09.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:49:09.691 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:49:09.745 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:49:10.872 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:49:10.894 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:49:10.894 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:49:10.895 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:49:10.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:49:11.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:49:11.150 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:49:11.391 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:49:11.392 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:49:11.431 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:49:11.461 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:49:11.498 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:49:11.499 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:49:11.536 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:49:12.692 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:51:12.740 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:51:12.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:51:13.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:51:13.896 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:51:13.896 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:51:13.896 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:51:13.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:51:14.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:51:14.144 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:51:14.395 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:51:14.395 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:51:14.426 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:51:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:51:14.492 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:51:14.493 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:51:14.533 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:51:15.563 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:53:15.623 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:53:15.661 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:53:16.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:53:16.833 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:53:16.833 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:53:16.833 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:53:16.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:53:17.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:53:17.094 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:53:17.318 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:53:17.319 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:53:17.358 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:53:17.388 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:53:17.424 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:53:17.426 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:53:17.464 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:53:18.537 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:55:18.607 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:55:18.646 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:55:20.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:55:20.287 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:55:20.287 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:55:20.287 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:55:20.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:55:20.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:55:20.531 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:55:20.769 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:55:20.769 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:55:20.808 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:55:20.838 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:55:20.874 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:55:20.876 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:55:20.915 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:55:22.155 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:57:22.207 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:57:22.245 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:57:24.154 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:57:24.177 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:57:24.177 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:57:24.177 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:57:24.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:57:24.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:57:24.421 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:57:24.683 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:57:24.687 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:57:24.724 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:57:24.760 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:57:24.792 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:57:24.792 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:57:24.834 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:57:27.255 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:59:27.306 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T09:59:27.442 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:59:29.677 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T09:59:29.694 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T09:59:29.694 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T09:59:29.694 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:59:29.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:59:29.912 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T09:59:29.956 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T09:59:30.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T09:59:30.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T09:59:30.226 INFO:teuthology.misc:Compressing logs... 2021-03-20T09:59:30.255 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T09:59:30.292 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T09:59:30.294 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T09:59:30.334 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T09:59:32.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:01:32.388 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:01:32.557 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:01:34.515 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:01:34.536 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:01:34.536 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:01:34.536 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:01:34.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:01:34.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:01:34.785 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:01:35.033 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:01:35.034 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:01:35.073 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:01:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:01:35.140 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:01:35.141 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:01:35.186 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:01:37.441 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:03:37.497 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:03:37.566 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:03:39.641 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:03:39.668 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:03:39.669 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:03:39.669 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:03:39.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:03:39.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:03:39.918 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:03:40.175 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:03:40.175 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:03:40.209 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:03:40.247 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:03:40.275 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:03:40.276 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:03:40.316 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:03:42.485 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:05:42.536 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:05:42.611 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:05:44.709 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:05:44.748 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:05:44.749 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:05:44.749 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:05:44.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:05:44.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:05:45.013 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:05:45.236 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:05:45.237 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:05:45.278 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:05:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:05:45.344 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:05:45.345 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:05:45.387 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:05:47.637 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:07:47.718 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:07:47.758 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:07:49.989 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:07:50.008 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:07:50.008 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:07:50.008 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:07:50.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:07:50.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:07:50.254 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:07:50.497 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:07:50.497 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:07:50.537 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:07:50.566 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:07:50.603 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:07:50.604 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:07:50.644 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:07:52.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:09:52.553 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:09:52.626 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:09:54.038 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:09:54.055 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:09:54.055 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:09:54.055 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:09:54.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:09:54.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:09:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:09:54.556 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:09:54.557 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:09:54.585 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:09:54.628 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:09:54.651 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:09:54.653 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:09:54.700 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:09:55.786 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:11:55.855 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:11:55.906 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:11:57.004 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:11:57.037 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:11:57.037 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:11:57.037 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:11:57.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:11:57.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:11:57.296 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:11:57.526 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:11:57.526 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:11:57.568 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:11:57.595 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:11:57.635 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:11:57.636 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:11:57.676 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:11:58.831 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:13:58.881 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:13:58.919 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:14:00.001 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:14:00.017 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:14:00.018 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:14:00.018 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:14:00.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:14:00.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:14:00.269 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:14:00.509 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:14:00.510 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:14:00.547 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:14:00.579 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:14:00.612 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:14:00.614 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:14:00.652 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:14:01.697 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:16:01.754 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:16:01.792 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:16:02.831 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:16:02.849 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:16:02.850 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:16:02.850 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:16:02.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:16:03.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:16:03.096 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:16:03.349 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:16:03.350 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:16:03.379 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:16:03.421 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:16:03.446 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:16:03.447 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:16:03.486 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:16:04.543 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:18:04.593 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:18:04.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:18:05.654 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:18:05.674 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:18:05.675 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:18:05.675 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:18:05.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:18:05.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:18:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:18:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:18:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:18:06.199 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:18:06.227 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:18:06.266 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:18:06.268 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:18:06.308 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:18:07.334 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:20:07.385 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:20:07.427 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:20:08.453 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:20:08.470 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:20:08.470 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:20:08.470 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:20:08.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:20:08.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:20:08.713 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:20:08.949 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:20:08.949 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:20:08.991 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:20:09.016 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:20:09.057 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:20:09.059 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:20:09.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:20:10.148 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:22:10.199 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:22:10.238 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:22:11.313 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:22:11.332 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:22:11.332 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:22:11.332 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:22:11.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:22:11.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:22:11.578 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:22:11.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:22:11.834 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:22:11.863 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:22:11.903 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:22:11.930 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:22:11.931 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:22:11.969 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:22:13.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:24:13.071 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:24:13.109 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:24:14.218 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:24:14.238 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:24:14.238 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:24:14.238 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:24:14.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:24:14.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:24:14.495 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:24:14.720 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:24:14.721 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:24:14.760 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:24:14.788 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:24:14.826 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:24:14.827 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:24:14.870 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:24:15.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:26:16.016 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:26:16.056 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:26:17.104 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:26:17.121 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:26:17.121 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:26:17.121 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:26:17.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:26:17.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:26:17.365 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:26:17.611 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:26:17.611 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:26:17.652 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:26:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:26:17.719 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:26:17.720 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:26:17.759 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:26:18.842 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:28:18.895 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:28:18.935 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:28:20.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:28:20.057 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:28:20.058 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:28:20.058 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:28:20.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:28:20.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:28:20.303 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:28:20.554 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:28:20.555 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:28:20.584 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:28:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:28:20.650 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:28:20.652 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:28:20.696 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:28:21.811 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:30:21.862 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:30:21.900 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:30:23.032 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:30:23.192 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:30:23.193 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:30:23.193 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:30:23.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:30:23.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:30:23.451 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:30:23.680 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:30:23.680 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:30:23.720 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:30:23.748 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:30:23.786 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:30:23.788 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:30:23.832 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:30:24.973 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:32:25.021 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:32:25.065 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:32:26.945 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:32:26.963 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:32:26.963 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:32:26.963 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:32:26.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:32:27.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:32:27.209 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:32:27.454 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:32:27.455 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:32:27.494 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:32:27.526 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:32:27.560 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:32:27.562 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:32:27.600 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:32:29.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:34:29.388 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:34:29.428 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:34:31.267 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:34:31.286 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:34:31.286 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:34:31.287 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:34:31.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:34:31.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:34:31.533 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:34:31.785 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:34:31.785 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:34:31.814 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:34:31.857 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:34:31.881 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:34:31.882 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:34:31.977 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:34:33.757 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:36:33.805 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:36:33.847 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:36:35.633 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:36:35.650 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:36:35.650 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:36:35.650 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:36:35.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:36:35.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:36:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:36:36.137 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:36:36.137 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:36:36.179 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:36:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:36:36.245 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:36:36.247 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:36:36.287 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:36:37.907 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:38:37.958 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:38:38.033 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:38:40.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:38:40.060 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:38:40.060 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:38:40.061 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:38:40.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:38:40.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:38:40.309 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:38:40.546 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:38:40.547 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:38:40.585 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:38:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:38:40.652 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:38:40.653 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:38:40.696 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:38:42.564 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:40:42.613 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:40:42.921 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:40:44.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:40:44.829 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:40:44.830 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:40:44.830 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:40:44.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:40:45.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:40:45.073 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:40:45.357 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:40:45.397 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:40:45.424 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:40:45.425 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:40:45.465 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:40:46.575 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:42:46.635 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:42:46.683 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:42:48.534 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:42:48.875 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:42:48.875 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:42:48.875 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:42:48.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:42:49.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:42:49.139 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:42:49.405 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:42:49.405 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:42:49.458 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:42:49.501 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:42:49.536 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:42:49.537 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:42:49.576 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:42:51.422 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:44:51.471 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:44:51.533 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:44:53.698 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:44:53.719 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:44:53.719 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:44:53.719 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:44:53.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:44:54.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:44:54.044 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:44:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:44:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:44:54.326 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:44:54.358 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:44:54.393 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:44:54.394 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:44:54.431 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:44:56.228 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:46:56.296 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:46:56.334 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:46:57.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:46:57.369 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:46:57.370 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:46:57.370 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:46:57.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:46:57.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:46:57.615 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:46:57.864 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:46:57.864 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:46:57.894 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:46:57.934 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:46:57.960 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:46:57.962 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:46:58.010 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:46:59.129 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:48:59.193 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:48:59.233 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:49:01.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:49:01.258 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:49:01.259 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:49:01.259 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:49:01.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:49:01.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:49:01.521 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:49:01.747 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:49:01.747 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:49:01.788 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:49:01.815 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:49:01.855 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:49:01.857 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:49:01.901 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:49:03.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:51:03.839 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:51:03.984 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:51:05.994 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:51:06.012 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:51:06.012 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:51:06.012 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:51:06.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:51:06.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:51:06.261 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:51:06.504 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:51:06.505 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:51:06.545 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:51:06.575 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:51:06.612 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:51:06.614 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:51:06.652 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:51:07.729 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:53:07.780 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:53:07.818 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:53:09.474 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:53:09.494 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:53:09.494 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:53:09.494 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:53:09.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:53:09.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:53:09.749 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:53:10.004 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:53:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:53:10.038 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:53:10.075 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:53:10.106 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:53:10.106 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:53:10.147 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:53:11.290 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:55:11.405 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:55:11.444 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:55:12.548 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:55:12.565 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:55:12.565 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:55:12.566 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:55:12.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:55:12.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:55:12.823 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:55:13.055 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:55:13.055 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:55:13.095 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:55:13.123 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:55:13.162 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:55:13.164 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:55:13.204 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:55:14.286 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:57:14.336 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:57:14.377 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:57:16.118 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:57:16.135 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:57:16.135 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:57:16.135 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:57:16.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:57:16.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:57:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:57:16.623 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:57:16.623 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:57:16.662 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:57:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:57:16.728 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:57:16.730 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:57:16.773 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:57:18.658 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:59:18.708 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T10:59:18.749 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:59:20.617 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T10:59:20.634 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T10:59:20.635 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T10:59:20.635 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:59:20.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:59:20.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T10:59:20.880 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T10:59:21.137 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T10:59:21.137 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T10:59:21.167 INFO:teuthology.misc:Compressing logs... 2021-03-20T10:59:21.207 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T10:59:21.234 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T10:59:21.235 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T10:59:21.274 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T10:59:22.348 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:01:22.396 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:01:22.447 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:01:23.605 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:01:23.622 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:01:23.622 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:01:23.622 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:01:23.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:01:23.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:01:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:01:24.112 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:01:24.112 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:01:24.152 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:01:24.180 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:01:24.219 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:01:24.220 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:01:24.261 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:01:25.385 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:03:25.444 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:03:25.482 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:03:26.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:03:26.630 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:03:26.630 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:03:26.630 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:03:26.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:03:26.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:03:26.911 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:03:27.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:03:27.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:03:27.314 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:03:27.343 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:03:27.382 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:03:27.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:03:27.421 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:03:28.475 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:05:28.526 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:05:28.564 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:05:29.660 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:05:29.678 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:05:29.678 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:05:29.678 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:05:29.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:05:29.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:05:29.928 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:05:30.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:05:30.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:05:30.217 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:05:30.252 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:05:30.283 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:05:30.285 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:05:30.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:05:31.439 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:07:31.488 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:07:31.525 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:07:32.569 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:07:32.585 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:07:32.586 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:07:32.586 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:07:32.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:07:32.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:07:32.848 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:07:33.076 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:07:33.076 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:07:33.114 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:07:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:07:33.180 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:07:33.181 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:07:33.222 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:07:34.290 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:09:34.353 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:09:34.393 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:09:35.450 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:09:35.520 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:09:35.520 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:09:35.520 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:09:35.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:09:35.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:09:35.764 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:09:36.008 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:09:36.008 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:09:36.047 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:09:36.080 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:09:36.113 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:09:36.115 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:09:36.156 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:09:37.271 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:11:37.318 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:11:37.356 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:11:38.386 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:11:38.402 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:11:38.403 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:11:38.403 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:11:38.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:11:38.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:11:38.673 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:11:38.953 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:11:38.954 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:11:38.983 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:11:39.082 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:11:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:11:39.086 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:11:39.124 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:11:40.223 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:13:40.285 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:13:40.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:13:42.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:13:42.158 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:13:42.159 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:13:42.159 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:13:42.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:13:42.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:13:42.417 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:13:42.639 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:13:42.639 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:13:42.683 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:13:42.708 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:13:42.749 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:13:42.751 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:13:42.793 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:13:43.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:15:43.900 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:15:43.938 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:15:45.851 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:15:45.874 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:15:45.874 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:15:45.874 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:15:45.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:15:46.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:15:46.120 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:15:46.362 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:15:46.530 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:15:46.567 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:15:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:15:46.636 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:15:46.637 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:15:46.681 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:15:48.585 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:17:48.640 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:17:48.712 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:17:50.764 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:17:50.786 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:17:50.786 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:17:50.787 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:17:50.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:17:51.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:17:51.041 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:17:51.295 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:17:51.295 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:17:51.331 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:17:51.365 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:17:51.398 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:17:51.399 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:17:51.440 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:17:53.394 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:19:53.446 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:19:53.485 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:19:54.546 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:19:54.563 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:19:54.564 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:19:54.564 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:19:54.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:19:54.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:19:54.829 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:19:55.055 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:19:55.055 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:19:55.096 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:19:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:19:55.163 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:19:55.164 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:19:55.203 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:19:56.272 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:21:56.323 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:21:56.363 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:21:57.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:21:57.525 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:21:57.525 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:21:57.526 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:21:57.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:21:57.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:21:57.770 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:21:58.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:21:58.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:21:58.053 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:21:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:21:58.118 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:21:58.120 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:21:58.157 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:21:59.257 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:23:59.309 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:23:59.349 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:24:00.534 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:24:00.551 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:24:00.551 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:24:00.552 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:24:00.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:24:00.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:24:00.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:24:01.051 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:24:01.051 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:24:01.082 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:24:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:24:01.148 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:24:01.150 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:24:01.189 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:24:02.242 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:26:02.289 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:26:02.338 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:26:03.410 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:26:03.428 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:26:03.429 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:26:03.429 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:26:03.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:26:03.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:26:03.695 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:26:03.964 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:26:03.964 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:26:04.003 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:26:04.035 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:26:04.072 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:26:04.073 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:26:04.118 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:26:05.219 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:28:05.269 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:28:05.367 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:28:07.365 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:28:07.387 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:28:07.387 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:28:07.387 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:28:07.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:28:07.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:28:07.632 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:28:07.876 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:28:07.876 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:28:07.913 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:28:07.946 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:28:07.980 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:28:07.981 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:28:08.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:28:10.169 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:30:10.247 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:30:10.286 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:30:12.018 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:30:12.037 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:30:12.037 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:30:12.037 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:30:12.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:30:12.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:30:12.287 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:30:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:30:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:30:12.574 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:30:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:30:12.640 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:30:12.650 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:30:12.696 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:30:14.032 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:32:14.080 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:32:14.124 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:32:15.282 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:32:15.298 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:32:15.298 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:32:15.299 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:32:15.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:32:15.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:32:15.554 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:32:15.788 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:32:15.788 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:32:15.827 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:32:15.855 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:32:15.894 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:32:15.896 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:32:15.937 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:32:17.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:34:17.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:34:17.169 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:34:19.060 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:34:19.100 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:34:19.101 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:34:19.101 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:34:19.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:34:19.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:34:19.358 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:34:19.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:34:19.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:34:19.644 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:34:19.674 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:34:19.710 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:34:19.712 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:34:19.750 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:34:21.438 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:36:21.491 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:36:21.538 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:36:22.618 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:36:22.635 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:36:22.635 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:36:22.635 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:36:22.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:36:22.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:36:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:36:23.133 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:36:23.133 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:36:23.164 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:36:23.202 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:36:23.230 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:36:23.231 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:36:23.278 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:36:24.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:38:24.454 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:38:24.501 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:38:25.567 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:38:25.584 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:38:25.585 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:38:25.585 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:38:25.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:38:25.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:38:25.843 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:38:26.067 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:38:26.067 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:38:26.109 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:38:26.135 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:38:26.174 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:38:26.176 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:38:26.213 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:38:27.276 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:40:27.330 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:40:27.369 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:40:28.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:40:28.511 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:40:28.512 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:40:28.512 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:40:28.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:40:28.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:40:28.753 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:40:28.994 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:40:28.994 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:40:29.032 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:40:29.064 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:40:29.099 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:40:29.101 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:40:29.140 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:40:30.226 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:42:30.276 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:42:30.313 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:42:31.384 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:42:31.401 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:42:31.402 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:42:31.402 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:42:31.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:42:31.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:42:31.645 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:42:31.904 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:42:31.904 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:42:31.932 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:42:31.976 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:42:31.998 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:42:31.999 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:42:32.037 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:42:33.105 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:44:33.157 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:44:33.268 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:44:35.090 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:44:35.107 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:44:35.107 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:44:35.107 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:44:35.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:44:35.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:44:35.364 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:44:35.589 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:44:35.590 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:44:35.628 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:44:35.659 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:44:35.695 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:44:35.697 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:44:35.740 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:44:37.279 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:46:37.326 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:46:37.407 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:46:39.596 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:46:39.614 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:46:39.614 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:46:39.614 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:46:39.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:46:39.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:46:39.863 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:46:40.102 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:46:40.103 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:46:40.143 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:46:40.173 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:46:40.210 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:46:40.211 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:46:40.265 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:46:41.339 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:48:41.387 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:48:41.428 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:48:43.415 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:48:43.444 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:48:43.445 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:48:43.445 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:48:43.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:48:43.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:48:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:48:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:48:44.022 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:48:44.051 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:48:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:48:44.118 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:48:44.119 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:48:44.162 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:48:45.223 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:50:45.273 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:50:45.315 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:50:46.819 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:50:46.837 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:50:46.837 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:50:46.837 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:50:46.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:50:47.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:50:47.224 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:50:47.465 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:50:47.465 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:50:47.511 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:50:47.545 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:50:47.581 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:50:47.582 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:50:47.635 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:50:48.687 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:52:48.739 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:52:48.775 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:52:49.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:52:49.873 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:52:49.873 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:52:49.874 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:52:49.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:52:50.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:52:50.120 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:52:50.366 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:52:50.367 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:52:50.404 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:52:50.437 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:52:50.470 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:52:50.472 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:52:50.508 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:52:51.630 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:54:51.682 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:54:51.749 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:54:52.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:54:52.857 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:54:52.857 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:54:52.858 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:54:52.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:54:53.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:54:53.104 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:54:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:54:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:54:53.395 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:54:53.433 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:54:53.462 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:54:53.463 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:54:53.500 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:54:54.620 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:56:54.677 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:56:54.748 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:56:55.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:56:55.870 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:56:55.870 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:56:55.870 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:56:55.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:56:56.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:56:56.131 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:56:56.359 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:56:56.360 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:56:56.400 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:56:56.430 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:56:56.467 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:56:56.468 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:56:56.505 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:56:57.572 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:58:57.625 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T11:58:57.662 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:58:58.712 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T11:58:58.731 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T11:58:58.731 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T11:58:58.731 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:58:58.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:58:58.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T11:58:58.978 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T11:58:59.220 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T11:58:59.220 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T11:58:59.259 INFO:teuthology.misc:Compressing logs... 2021-03-20T11:58:59.292 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T11:58:59.326 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T11:58:59.328 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T11:58:59.372 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T11:59:00.532 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:01:00.583 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:01:00.638 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:01:01.851 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:01:01.868 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:01:01.868 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:01:01.869 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:01:01.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:01:02.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:01:02.126 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:01:02.374 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:01:02.374 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:01:02.412 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:01:02.443 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:01:02.478 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:01:02.479 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:01:02.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:01:03.591 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:03:03.639 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:03:03.678 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:03:05.576 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:03:05.594 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:03:05.594 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:03:05.594 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:03:05.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:03:05.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:03:05.860 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:03:06.079 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:03:06.080 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:03:06.120 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:03:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:03:06.186 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:03:06.187 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:03:06.227 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:03:07.559 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:05:07.608 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:05:07.650 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:05:09.886 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:05:09.903 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:05:09.903 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:05:09.903 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:05:09.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:05:10.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:05:10.150 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:05:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:05:10.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:05:10.431 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:05:10.457 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:05:10.497 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:05:10.526 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T12:05:10.526 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:05:10.565 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T12:05:10.566 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:05:10.567 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:05:10.593 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T12:05:10.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T12:05:10.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:05:11.815 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:05:13.336 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:07:13.396 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:07:13.434 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:07:14.502 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:07:14.519 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:07:14.519 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:07:14.519 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:07:14.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:07:14.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:07:14.765 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:07:17.794 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T12:07:17.832 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:07:18.910 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:09:18.960 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:09:19.016 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:09:21.181 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:09:21.200 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:09:21.201 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:09:21.201 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:09:21.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:09:21.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:09:21.421 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T12:09:21.483 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:09:22.589 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:11:22.640 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:11:22.680 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:11:23.978 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:11:23.995 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:11:23.995 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:11:23.995 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:11:23.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:11:24.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:11:24.235 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:11:41.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T12:11:41.679 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:11:43.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:13:43.980 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:13:44.053 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:13:46.018 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:13:46.038 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:13:46.038 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:13:46.039 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:13:46.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:13:46.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:13:46.290 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:13:46.491 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:13:46.525 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:13:46.526 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:13:46.576 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:13:46.576 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:13:46.578 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:13:46.605 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:13:46.643 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:13:46.675 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T12:13:46.675 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:13:46.708 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T12:13:46.708 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:13:46.709 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:13:46.742 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T12:13:46.774 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T12:13:46.809 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T12:13:46.809 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:13:46.839 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T12:13:46.839 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:13:46.881 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:13:48.823 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:15:48.877 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:15:48.916 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:15:49.984 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:15:50.002 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:15:50.002 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:15:50.002 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:15:50.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:15:50.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:15:50.265 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:15:50.449 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:15:50.481 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:15:50.481 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:15:50.520 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:15:50.520 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:15:50.521 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:15:50.550 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:15:50.585 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:15:50.619 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T12:15:50.619 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:15:50.651 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T12:15:50.651 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:15:50.652 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:15:50.689 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T12:15:50.716 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T12:15:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T12:15:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:15:50.783 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T12:15:50.783 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:15:50.820 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:15:51.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:17:51.919 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:17:51.958 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:17:53.016 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:17:53.036 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:17:53.036 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:17:53.036 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:17:53.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:17:53.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:17:53.280 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:17:53.476 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:17:53.509 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:17:53.509 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:17:53.542 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:17:53.542 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:17:53.543 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:17:53.579 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:17:53.649 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T12:17:53.649 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:17:53.690 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:17:53.716 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T12:17:53.745 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T12:17:53.784 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T12:17:53.785 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:17:53.813 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T12:17:53.813 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:17:53.898 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:17:54.961 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:19:55.012 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:19:55.055 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:19:56.102 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:19:56.119 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:19:56.119 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:19:56.119 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:19:56.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:19:56.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:19:56.368 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:19:56.564 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:19:56.594 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:19:56.595 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:19:56.628 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:19:56.628 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:19:56.630 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:19:56.666 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:19:56.694 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:19:56.696 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:19:56.735 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:19:57.771 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:21:57.817 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:21:57.856 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:21:58.900 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:21:58.919 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:21:58.919 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:21:58.919 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:21:58.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:21:59.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:21:59.184 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:21:59.378 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:21:59.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:21:59.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:21:59.445 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:21:59.445 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:21:59.446 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:21:59.477 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:21:59.513 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:21:59.515 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:21:59.586 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:22:00.622 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:24:00.672 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:24:00.713 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:24:01.735 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:24:01.752 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:24:01.752 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:24:01.753 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:24:01.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:24:01.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:24:01.997 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:24:02.186 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:24:02.216 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:24:02.217 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:24:02.251 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:24:02.251 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:24:02.252 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:24:02.286 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:24:02.317 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:24:02.319 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:24:02.358 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:24:03.397 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:26:03.446 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:26:03.487 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:26:04.527 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:26:04.544 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:26:04.544 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:26:04.544 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:26:04.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:26:04.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:26:04.788 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:26:04.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:26:04.994 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:26:05.034 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:26:05.062 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:26:05.099 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:26:05.100 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:26:05.143 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:26:06.198 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:28:06.249 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:28:06.303 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:28:08.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:28:08.187 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:28:08.188 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:28:08.188 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:28:08.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:28:08.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:28:08.447 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:28:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:28:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:28:08.688 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:28:08.714 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:28:08.756 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:28:08.757 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:28:08.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:28:10.623 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:30:10.675 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:30:10.713 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:30:12.636 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:30:12.653 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:30:12.653 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:30:12.653 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:30:12.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:30:12.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:30:12.901 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:30:13.113 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:30:13.113 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:30:13.154 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:30:13.182 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:30:13.221 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:30:13.222 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:30:13.264 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:30:15.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:32:15.117 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:32:15.156 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:32:17.009 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:32:17.029 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:32:17.029 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:32:17.029 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:32:17.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:32:17.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:32:17.282 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:32:17.488 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:32:17.488 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:32:17.528 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:32:17.557 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:32:17.595 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:32:17.596 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:32:17.636 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:32:18.808 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:34:18.861 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:34:18.907 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:34:20.003 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:34:20.022 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:34:20.022 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:34:20.022 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:34:20.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:34:20.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:34:20.280 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:34:20.470 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:34:20.470 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:34:20.510 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:34:20.539 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:34:20.576 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:34:20.578 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:34:20.618 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:34:21.681 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:36:21.729 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:36:21.772 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:36:22.877 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:36:22.894 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:36:22.894 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:36:22.894 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:36:22.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:36:23.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:36:23.140 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:36:23.347 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:36:23.347 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:36:23.388 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:36:23.418 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:36:23.454 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:36:23.457 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:36:23.508 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:36:24.577 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:38:24.631 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:38:24.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:38:25.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:38:25.769 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:38:25.769 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:38:25.770 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:38:25.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:38:25.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:38:26.016 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:38:26.225 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:38:26.225 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:38:26.266 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:38:26.294 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:38:26.334 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:38:26.336 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:38:26.372 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:38:27.415 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:40:27.464 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:40:27.509 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:40:28.561 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:40:28.585 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:40:28.585 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:40:28.585 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:40:28.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:40:28.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:40:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:40:47.298 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T12:40:47.335 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:40:48.402 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:42:48.453 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:42:48.492 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:42:49.551 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:42:49.568 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:42:49.568 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:42:49.568 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:42:49.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:42:49.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:42:49.815 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:42:49.847 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T12:42:50.943 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T12:42:52.040 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T12:42:53.137 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T12:42:54.234 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T12:42:55.334 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T12:42:56.392 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T12:42:57.488 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T12:42:58.586 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T12:42:59.683 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T12:42:59.684 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T12:42:59.721 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:43:00.781 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:45:00.831 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:45:00.868 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:45:01.901 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:45:01.918 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:45:01.918 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:45:01.918 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:45:01.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:45:02.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:45:02.161 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:45:05.834 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:45:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:45:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:45:05.952 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:45:05.952 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:45:05.954 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:45:05.989 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:45:05.989 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:45:06.061 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T12:45:06.061 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:45:06.086 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T12:45:06.086 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:45:06.087 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:45:06.131 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T12:45:06.149 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T12:45:06.200 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T12:45:06.201 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:45:06.225 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T12:45:06.225 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:45:06.262 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:45:07.375 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:47:07.425 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:47:07.466 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:47:08.532 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:47:08.549 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:47:08.549 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:47:08.549 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:47:08.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:47:08.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:47:08.811 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:47:08.988 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:47:09.018 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:47:09.019 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:47:09.056 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:47:09.056 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:47:09.057 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:47:09.090 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:47:09.158 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T12:47:09.158 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:47:09.206 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:47:09.232 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T12:47:09.259 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T12:47:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T12:47:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:47:09.325 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T12:47:09.326 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:47:09.364 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:47:10.457 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:49:10.514 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:49:10.552 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:49:11.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:49:11.617 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:49:11.617 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:49:11.617 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:49:11.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:49:11.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:49:11.860 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:49:12.061 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:49:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:49:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:49:12.126 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:49:12.126 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:49:12.129 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:49:12.159 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:49:12.194 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:49:12.195 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:49:12.232 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:49:13.265 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:51:13.318 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:51:13.359 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:51:14.518 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:51:14.536 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:51:14.536 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:51:14.536 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:51:14.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:51:14.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:51:14.779 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:51:14.983 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:51:15.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:51:15.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:51:15.049 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:51:15.049 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:51:15.051 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:51:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:51:15.117 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:51:15.118 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:51:15.156 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:51:16.266 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:53:16.323 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:53:16.418 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:53:17.532 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:53:17.552 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:53:17.552 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:53:17.552 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:53:17.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:53:17.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:53:17.806 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:53:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:53:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:53:18.063 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:53:18.091 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:53:18.127 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:53:18.129 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:53:18.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:53:19.237 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:55:19.288 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:55:19.326 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:55:20.444 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:55:20.465 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:55:20.465 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:55:20.465 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:55:20.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:55:20.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:55:20.706 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:55:20.949 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:55:20.950 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:55:20.988 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:55:21.020 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:55:21.054 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:55:21.056 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:55:21.095 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:55:22.167 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:57:22.214 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:57:22.259 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:57:24.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:57:24.357 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:57:24.358 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:57:24.358 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:57:24.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:57:24.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:57:24.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:57:24.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:57:24.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:57:24.882 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:57:24.914 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:57:24.949 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:57:24.951 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:57:24.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:57:26.331 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:59:26.385 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T12:59:26.424 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:59:27.658 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T12:59:27.675 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T12:59:27.675 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T12:59:27.675 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:59:27.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:59:27.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T12:59:27.934 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T12:59:28.142 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T12:59:28.143 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T12:59:28.183 INFO:teuthology.misc:Compressing logs... 2021-03-20T12:59:28.215 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T12:59:28.248 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T12:59:28.249 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T12:59:28.296 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T12:59:29.374 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:01:29.424 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:01:29.462 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:01:30.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:01:30.660 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:01:30.660 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:01:30.661 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:01:30.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:01:30.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:01:30.907 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:01:31.153 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:01:31.153 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:01:31.193 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:01:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:01:31.260 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T13:01:31.261 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T13:01:31.300 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:01:32.427 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:03:32.488 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:03:32.526 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:03:34.083 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:03:34.101 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:03:34.101 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:03:34.101 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:03:34.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:03:34.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:03:34.347 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:03:52.770 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T13:03:52.829 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:03:53.944 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:05:53.993 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:05:54.039 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:05:55.143 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:05:55.162 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:05:55.162 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:05:55.162 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:05:55.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:05:55.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:05:55.419 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:05:55.480 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T13:05:56.537 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T13:05:57.634 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T13:05:58.734 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T13:05:59.834 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T13:06:00.935 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T13:06:02.033 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T13:06:03.092 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T13:06:04.191 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T13:06:05.288 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T13:06:05.288 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T13:06:05.335 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:06:06.399 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:08:06.451 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:08:06.489 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:08:07.603 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:08:07.619 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:08:07.620 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:08:07.620 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:08:07.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:08:07.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:08:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:08:39.808 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T13:08:39.841 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:08:39.842 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:08:39.867 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:08:39.867 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:08:39.869 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:08:39.912 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:08:39.923 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T13:08:39.982 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:08:39.982 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:08:40.002 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:08:40.002 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:08:40.004 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:08:40.051 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:08:40.058 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T13:08:40.119 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:08:40.120 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:08:40.136 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:08:40.136 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:08:40.175 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:08:41.245 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:10:41.295 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:10:41.366 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:10:42.432 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:10:42.449 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:10:42.449 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:10:42.449 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:10:42.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:10:42.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:10:42.697 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:10:42.991 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T13:10:43.022 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:10:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:10:43.050 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:10:43.050 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:10:43.052 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:10:43.092 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:10:43.105 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-20T13:10:43.162 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:10:43.162 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:10:43.178 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:10:43.179 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:10:43.180 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:10:43.229 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:10:43.233 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T13:10:43.296 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:10:43.296 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:10:43.310 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:10:43.311 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:10:43.351 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:10:44.373 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:12:44.425 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:12:44.465 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:12:45.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:12:45.558 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:12:45.559 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:12:45.559 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:12:45.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:12:45.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:12:45.822 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:13:04.199 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T13:13:04.237 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:13:05.287 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:15:05.377 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:15:05.417 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:15:06.455 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:15:06.474 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:15:06.474 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:15:06.474 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:15:06.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:15:06.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:15:06.728 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:15:07.021 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-20T13:15:07.052 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:15:07.052 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:15:07.083 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:15:07.083 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:15:07.085 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:15:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:15:07.194 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:15:07.194 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:15:07.229 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:15:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:15:07.270 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-20T13:15:07.333 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:15:07.333 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:15:07.350 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:15:07.350 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:15:07.391 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:15:08.487 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:17:08.536 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:17:08.578 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:17:09.689 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:17:09.707 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:17:09.707 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:17:09.707 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:17:09.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:17:09.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:17:09.962 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:17:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:17:10.296 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:17:10.325 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:17:10.366 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:17:10.436 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:17:10.436 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:17:10.477 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:17:10.507 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:17:10.576 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:17:10.576 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:17:10.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:17:11.817 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:19:11.912 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:19:11.952 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:19:13.754 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:19:13.772 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:19:13.772 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:19:13.772 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:19:13.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:19:13.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:19:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:19:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:19:14.352 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:19:14.383 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:19:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:19:14.488 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:19:14.488 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:19:14.531 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:19:14.557 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:19:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:19:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:19:14.694 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:19:15.692 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:21:15.740 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:21:15.776 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:21:16.987 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:21:17.004 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:21:17.004 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:21:17.004 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:21:17.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:21:17.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:21:17.257 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:21:17.593 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:21:17.593 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:21:17.625 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:21:17.666 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:21:17.736 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:21:17.736 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:21:18.154 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:21:18.184 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:21:18.254 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:21:18.255 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:21:18.402 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:21:19.437 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:23:19.500 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:23:19.565 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:23:20.596 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:23:20.616 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:23:20.616 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:23:20.616 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:23:20.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:23:20.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:23:20.867 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:23:21.205 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:23:21.205 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:23:21.236 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:23:21.275 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:23:21.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:23:21.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:23:24.602 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:23:24.641 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:23:24.713 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:23:24.713 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:23:25.594 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:23:26.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:25:26.749 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:25:26.790 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:25:27.850 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:25:27.866 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:25:27.866 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:25:27.866 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:25:27.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:25:28.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:25:28.123 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:25:28.445 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:25:28.446 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:25:28.476 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:25:28.516 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:25:28.586 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:25:28.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:25:28.653 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:25:28.681 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:25:28.751 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:25:28.751 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:25:30.602 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:25:31.641 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:27:31.692 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:27:31.728 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:27:32.743 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:27:32.760 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:27:32.760 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:27:32.760 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:27:32.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:27:32.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:27:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:27:33.350 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:27:33.350 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:27:33.380 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:27:33.420 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:27:33.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:27:33.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:27:33.544 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:27:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:27:33.642 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:27:33.643 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:27:36.059 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:27:37.252 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:29:37.314 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:29:37.351 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:29:38.374 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:29:38.391 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:29:38.391 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:29:38.391 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:29:38.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:29:38.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:29:38.635 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:29:56.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T13:29:56.969 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:29:58.032 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:31:58.081 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:31:58.121 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:31:59.213 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:31:59.230 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:31:59.231 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:31:59.231 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:31:59.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:31:59.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:31:59.497 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:32:17.987 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T13:32:18.026 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:32:19.132 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:34:19.180 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:34:19.222 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:34:20.272 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:34:20.288 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:34:20.288 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:34:20.288 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:34:20.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:34:20.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:34:20.577 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:34:20.781 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:34:20.812 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:34:20.812 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:34:20.865 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:34:20.865 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:34:20.866 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:34:20.894 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:34:20.931 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:34:20.964 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:34:20.964 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:34:20.998 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:34:20.998 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:34:20.999 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:34:21.032 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:34:21.063 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:34:21.100 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:34:21.100 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:34:21.131 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:34:21.131 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:34:21.175 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:34:22.307 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:36:22.359 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:36:22.396 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:36:23.464 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:36:23.482 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:36:23.482 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:36:23.482 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:36:23.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:36:23.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:36:23.724 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:36:23.913 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:36:23.943 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:36:23.943 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:36:23.978 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:36:23.978 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:36:23.979 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:36:24.012 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:36:24.043 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:36:24.080 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:36:24.081 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:36:24.108 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:36:24.109 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:36:24.110 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:36:24.149 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:36:24.174 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:36:24.216 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:36:24.217 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:36:24.242 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:36:24.242 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:36:24.281 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:36:25.363 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:38:25.425 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:38:25.467 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:38:26.535 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:38:26.554 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:38:26.554 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:38:26.554 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:38:26.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:38:26.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:38:26.810 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:38:32.866 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T13:38:32.905 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:38:33.943 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:40:33.997 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:40:34.039 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:40:35.115 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:40:35.132 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:40:35.132 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:40:35.132 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:40:35.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:40:35.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:40:35.376 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:40:35.577 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:40:35.607 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:40:35.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:40:35.655 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:40:35.656 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:40:35.657 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:40:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:40:35.755 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T13:40:35.755 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:40:35.812 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:40:35.839 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:40:35.861 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T13:40:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:40:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:40:35.933 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T13:40:35.933 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:40:35.972 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:40:37.029 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:42:37.083 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:42:37.123 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:42:38.226 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:42:38.243 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:42:38.243 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:42:38.244 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:42:38.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:42:38.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:42:38.489 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:42:38.692 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:42:38.722 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:42:38.722 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:42:38.759 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:42:38.759 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:42:38.761 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:42:38.793 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:42:38.827 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T13:42:38.828 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T13:42:38.885 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:42:39.958 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:44:40.010 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:44:40.047 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:44:41.094 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:44:41.111 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:44:41.111 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:44:41.111 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:44:41.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:44:41.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:44:41.373 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:44:41.589 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:44:41.589 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:44:41.627 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:44:41.658 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:44:41.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T13:44:41.695 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T13:44:41.731 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:44:42.740 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:46:42.787 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:46:42.909 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:46:44.053 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:46:44.070 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:46:44.070 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:46:44.070 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:46:44.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:46:44.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:46:44.319 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:46:44.919 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:46:44.919 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:46:44.953 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:46:45.059 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:46:45.061 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T13:46:45.063 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T13:46:45.099 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:46:46.253 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:48:46.305 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:48:46.343 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:48:47.382 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:48:47.398 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:48:47.398 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:48:47.398 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:48:47.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:48:47.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:48:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:48:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:48:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:48:47.919 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:48:47.949 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:48:47.984 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T13:48:47.985 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T13:48:48.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:48:49.061 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:50:49.113 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:50:49.150 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:50:50.937 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:50:50.959 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:50:50.959 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:50:50.959 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:50:50.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:50:51.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:50:51.225 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:50:51.460 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:50:51.461 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:50:51.490 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:50:51.530 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:50:51.557 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T13:50:51.558 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T13:50:51.601 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:50:52.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:52:52.742 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:52:52.781 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:52:54.561 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:52:54.578 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:52:54.578 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:52:54.578 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:52:54.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:52:54.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:52:54.826 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:52:55.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:52:55.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:52:55.098 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:52:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:52:55.164 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T13:52:55.166 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T13:52:55.205 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:52:56.252 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:54:56.301 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:54:56.338 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:54:57.352 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:54:57.368 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:54:57.368 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:54:57.369 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:54:57.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:54:57.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:54:57.618 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:54:57.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:54:57.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:54:57.897 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:54:57.925 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:54:57.964 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T13:54:57.965 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T13:54:58.001 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:54:59.061 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:56:59.111 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:56:59.148 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:57:00.125 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:57:00.144 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:57:00.145 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:57:00.145 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:57:00.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:57:00.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:57:00.409 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:57:00.639 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:57:00.639 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:57:00.667 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:57:00.709 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:57:00.733 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T13:57:00.735 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T13:57:00.770 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:57:01.813 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:59:01.867 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T13:59:01.912 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:59:02.947 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T13:59:02.963 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T13:59:02.964 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T13:59:02.964 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:59:02.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:59:03.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T13:59:03.209 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T13:59:03.436 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T13:59:03.436 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T13:59:03.475 INFO:teuthology.misc:Compressing logs... 2021-03-20T13:59:03.506 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T13:59:03.541 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T13:59:03.543 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T13:59:03.579 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T13:59:04.625 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:01:04.675 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:01:04.719 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:01:05.740 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:01:05.757 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:01:05.757 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:01:05.757 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:01:05.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:01:05.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:01:06.003 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:01:06.251 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:01:06.252 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:01:06.290 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:01:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:01:06.356 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:01:06.358 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:01:06.396 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:01:07.432 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:03:07.495 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:03:07.535 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:03:08.854 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:03:08.870 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:03:08.871 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:03:08.871 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:03:08.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:03:09.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:03:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:03:09.411 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:03:09.411 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:03:09.446 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:03:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:03:09.512 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:03:09.513 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:03:09.598 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:03:10.640 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:05:10.686 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:05:10.722 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:05:11.818 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:05:11.838 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:05:11.838 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:05:11.838 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:05:11.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:05:12.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:05:12.084 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:05:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:05:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:05:12.353 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:05:12.385 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:05:12.417 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:05:12.418 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:05:12.456 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:05:13.505 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:07:13.556 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:07:13.597 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:07:14.615 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:07:14.632 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:07:14.632 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:07:14.632 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:07:14.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:07:14.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:07:14.876 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:07:15.105 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:07:15.106 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:07:15.146 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:07:15.174 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:07:15.212 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:07:15.213 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:07:15.250 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:07:16.258 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:09:16.309 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:09:16.349 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:09:17.396 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:09:17.413 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:09:17.413 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:09:17.413 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:09:17.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:09:17.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:09:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:09:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:09:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:09:17.945 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:09:17.976 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:09:18.010 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:09:18.011 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:09:18.048 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:09:19.066 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:11:19.114 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:11:19.152 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:11:20.220 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:11:20.237 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:11:20.237 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:11:20.237 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:11:20.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:11:20.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:11:20.483 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:11:20.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:11:20.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:11:20.750 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:11:20.781 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:11:20.816 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:11:20.818 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:11:20.856 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:11:21.894 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:13:21.946 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:13:21.986 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:13:23.050 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:13:23.066 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:13:23.067 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:13:23.067 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:13:23.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:13:23.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:13:23.309 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:13:23.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:13:23.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:13:23.576 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:13:23.607 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:13:23.642 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:13:23.643 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:13:23.680 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:13:24.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:15:24.769 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:15:24.807 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:15:25.831 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:15:25.846 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:15:25.847 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:15:25.847 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:15:25.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:15:26.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:15:26.105 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:15:26.343 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:15:26.343 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:15:26.371 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:15:26.414 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:15:26.437 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:15:26.438 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:15:26.474 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:15:27.556 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:17:27.605 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:17:27.644 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:17:28.675 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:17:28.691 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:17:28.692 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:17:28.692 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:17:28.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:17:28.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:17:28.934 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:17:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:17:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:17:29.202 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:17:29.233 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:17:29.268 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:17:29.269 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:17:29.306 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:17:30.295 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:19:30.342 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:19:30.383 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:19:31.462 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:19:31.479 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:19:31.479 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:19:31.479 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:19:31.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:19:31.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:19:31.724 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:19:31.958 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:19:31.959 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:19:31.996 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:19:32.029 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:19:32.061 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:19:32.062 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:19:32.106 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:19:33.265 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:21:33.313 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:21:33.351 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:21:34.401 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:21:34.419 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:21:34.419 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:21:34.419 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:21:34.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:21:34.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:21:34.685 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:21:34.922 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:21:34.922 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:21:34.951 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:21:34.992 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:21:35.016 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:21:35.018 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:21:35.060 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:21:36.117 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:23:36.288 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:23:36.325 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:23:37.351 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:23:37.368 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:23:37.369 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:23:37.369 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:23:37.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:23:37.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:23:37.613 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:23:37.853 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:23:37.853 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:23:37.892 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:23:37.922 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:23:37.958 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:23:37.960 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:23:37.996 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:23:38.987 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:25:39.036 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:25:39.074 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:25:40.111 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:25:40.128 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:25:40.128 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:25:40.128 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:25:40.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:25:40.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:25:40.372 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:25:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:25:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:25:40.640 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:25:40.672 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:25:40.707 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:25:40.708 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:25:40.746 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:25:41.814 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:27:41.862 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:27:41.913 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:27:42.974 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:27:42.991 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:27:42.991 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:27:42.991 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:27:42.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:27:43.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:27:43.248 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:27:43.477 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:27:43.477 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:27:43.505 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:27:43.546 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:27:43.572 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:27:43.573 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:27:43.610 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:27:44.616 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:29:44.668 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:29:44.709 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:29:45.790 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:29:45.806 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:29:45.806 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:29:45.806 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:29:45.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:29:46.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:29:46.050 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:29:46.280 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:29:46.281 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:29:46.321 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:29:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:29:46.386 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:29:46.388 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:29:46.425 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:29:47.477 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:31:47.524 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:31:47.563 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:31:48.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:31:48.608 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:31:48.608 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:31:48.608 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:31:48.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:31:48.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:31:48.852 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:31:49.080 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:31:49.081 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:31:49.118 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:31:49.151 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:31:49.182 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:31:49.184 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:31:49.220 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:31:50.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:33:50.280 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:33:50.320 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:33:51.597 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:33:51.614 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:33:51.614 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:33:51.614 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:33:51.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:33:51.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:33:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:33:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:33:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:33:52.221 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:33:52.250 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:33:52.287 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:33:52.289 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:33:52.338 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:33:53.389 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:35:53.443 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:35:53.480 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:35:54.520 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:35:54.536 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:35:54.536 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:35:54.536 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:35:54.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:35:54.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:35:54.778 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:35:55.010 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:35:55.010 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:35:55.051 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:35:55.079 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:35:55.116 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:35:55.117 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:35:55.154 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:35:56.233 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:37:56.280 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:37:56.358 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:37:57.480 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:37:57.497 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:37:57.497 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:37:57.497 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:37:57.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:37:57.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:37:57.746 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:37:57.973 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:37:57.974 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:37:58.012 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:37:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:37:58.078 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:37:58.079 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:37:58.116 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:37:59.175 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:39:59.221 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:39:59.259 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:40:00.304 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:40:00.321 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:40:00.321 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:40:00.321 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:40:00.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:40:00.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:40:00.580 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:40:00.812 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:40:00.812 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:40:00.853 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:40:00.881 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:40:00.919 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:40:00.920 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:40:00.958 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:40:02.016 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:42:02.068 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:42:02.104 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:42:03.145 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:42:03.162 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:42:03.162 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:42:03.162 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:42:03.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:42:03.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:42:03.410 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:42:03.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:42:03.637 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:42:03.675 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:42:03.707 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:42:03.740 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:42:03.742 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:42:03.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:42:04.795 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:44:04.845 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:44:04.885 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:44:06.686 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:44:06.703 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:44:06.703 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:44:06.703 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:44:06.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:44:06.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:44:06.952 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:44:07.196 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:44:07.196 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:44:07.235 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:44:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:44:07.301 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:44:07.302 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:44:07.343 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:44:08.382 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:46:08.430 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:46:08.466 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:46:09.505 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:46:09.522 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:46:09.522 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:46:09.522 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:46:09.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:46:09.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:46:09.780 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:46:10.018 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:46:10.018 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:46:10.046 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:46:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:46:10.114 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:46:10.116 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:46:10.155 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:46:11.192 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:48:11.248 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:48:11.286 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:48:12.352 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:48:12.370 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:48:12.370 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:48:12.370 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:48:12.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:48:12.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:48:12.619 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:48:12.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:48:12.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:48:12.886 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:48:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:48:12.950 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:48:12.952 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:48:12.994 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:48:14.037 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:50:14.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:50:14.134 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:50:15.185 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:50:15.202 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:50:15.203 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:50:15.203 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:50:15.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:50:15.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:50:15.443 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:50:15.674 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:50:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:50:15.717 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:50:15.751 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:50:15.783 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:50:15.785 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:50:15.840 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:50:16.864 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:52:16.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:52:16.953 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:52:17.982 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:52:17.999 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:52:18.000 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:52:18.000 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:52:18.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:52:18.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:52:18.265 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:52:18.505 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:52:18.505 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:52:18.535 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:52:18.576 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:52:18.601 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:52:18.602 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:52:18.640 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:52:19.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:54:19.743 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:54:19.782 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:54:20.875 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:54:20.892 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:54:20.892 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:54:20.893 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:54:20.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:54:21.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:54:21.140 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:54:21.366 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:54:21.366 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:54:21.404 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:54:21.435 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:54:21.468 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:54:21.469 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:54:21.507 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:54:22.548 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:56:22.601 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:56:22.640 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:56:23.646 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:56:23.663 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:56:23.663 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:56:23.663 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:56:23.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:56:23.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:56:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:56:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:56:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:56:24.188 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:56:24.214 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:56:24.255 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:56:24.257 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:56:24.299 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:56:25.384 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:58:25.434 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T14:58:25.473 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:58:26.513 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T14:58:26.530 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T14:58:26.530 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T14:58:26.530 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:58:26.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:58:26.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T14:58:26.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T14:58:27.033 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T14:58:27.033 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T14:58:27.062 INFO:teuthology.misc:Compressing logs... 2021-03-20T14:58:27.104 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T14:58:27.129 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T14:58:27.131 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T14:58:27.175 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T14:58:28.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:00:28.280 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:00:28.319 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:00:29.461 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:00:29.494 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:00:29.494 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:00:29.494 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:00:29.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:00:29.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:00:29.759 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:00:30.066 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:00:30.067 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:00:30.107 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:00:30.145 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:00:30.175 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:00:30.177 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:00:30.216 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:00:31.356 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:02:31.407 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:02:31.445 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:02:32.439 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:02:32.456 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:02:32.456 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:02:32.456 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:02:32.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:02:32.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:02:32.699 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:02:51.075 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T15:02:51.140 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:02:52.203 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:04:52.255 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:04:52.290 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:04:53.364 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:04:53.380 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:04:53.380 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:04:53.380 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:04:53.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:04:53.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:04:53.637 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:04:53.650 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T15:04:54.747 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T15:04:55.843 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T15:04:56.939 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T15:04:58.042 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-20T15:05:59.044 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T15:05:59.080 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:06:00.222 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:08:00.292 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:08:00.328 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:08:01.338 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:08:01.354 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:08:01.355 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:08:01.355 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:08:01.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:08:01.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:08:01.603 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:08:01.800 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:08:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:08:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:08:01.879 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:08:01.879 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:08:01.880 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:08:01.909 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:08:01.946 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:08:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T15:08:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:08:02.011 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T15:08:02.012 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:08:02.013 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:08:02.048 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T15:08:02.077 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T15:08:02.116 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T15:08:02.116 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:08:02.142 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T15:08:02.142 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:08:02.179 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:08:03.192 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:10:03.257 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:10:03.299 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:10:04.348 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:10:04.364 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:10:04.365 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:10:04.365 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:10:04.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:10:04.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:10:04.622 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:11:04.581 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-20T15:11:04.622 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:11:05.696 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:13:05.764 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:13:05.801 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:13:06.949 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:13:06.984 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:13:06.984 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:13:06.985 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:13:06.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:13:07.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:13:07.254 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:13:07.522 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:13:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:13:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:13:07.597 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:13:07.598 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:13:07.598 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:13:07.630 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:13:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-20T15:13:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:13:07.763 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:13:07.789 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T15:13:07.807 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-20T15:13:07.859 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T15:13:07.860 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:13:07.888 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-20T15:13:07.888 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:13:07.971 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:13:09.051 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:15:09.098 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:15:09.136 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:15:10.213 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:15:10.235 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:15:10.235 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:15:10.235 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:15:10.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:15:10.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:15:10.546 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:15:10.733 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:15:10.762 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:15:10.762 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:15:10.800 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:15:10.800 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:15:10.801 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:15:10.831 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:15:10.868 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:15:10.871 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:15:10.907 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:15:11.931 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:17:11.985 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:17:12.028 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:17:13.102 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:17:13.156 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:17:13.156 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:17:13.157 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:17:13.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:17:13.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:17:13.401 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:17:13.614 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:17:13.644 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:17:13.644 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:17:13.682 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:17:13.682 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:17:13.683 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:17:13.712 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:17:13.749 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:17:13.751 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:17:13.824 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:17:14.877 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:19:14.930 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:19:14.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:19:16.676 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:19:16.695 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:19:16.695 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:19:16.695 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:19:16.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:19:16.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:19:16.940 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:19:17.165 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:19:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:19:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:19:17.225 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:19:17.225 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:19:17.227 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:19:17.268 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:19:17.294 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:19:17.296 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:19:17.335 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:19:18.314 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:21:18.360 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:21:18.398 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:21:19.400 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:21:19.417 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:21:19.418 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:21:19.418 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:21:19.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:21:19.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:21:19.685 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:21:19.879 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:21:19.911 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:21:19.911 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:21:19.947 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:21:19.947 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:21:19.949 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:21:19.981 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:21:20.016 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:21:20.017 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:21:20.052 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:21:21.037 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:23:21.086 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:23:21.121 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:23:22.076 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:23:22.094 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:23:22.094 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:23:22.095 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:23:22.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:23:22.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:23:22.336 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:23:22.550 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:23:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:23:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:23:22.619 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:23:22.619 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:23:22.621 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:23:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:23:22.688 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:23:22.689 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:23:22.722 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:23:23.676 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:25:23.723 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:25:23.760 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:25:24.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:25:24.854 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:25:24.854 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:25:24.854 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:25:24.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:25:25.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:25:25.095 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:25:25.307 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:25:25.336 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:25:25.336 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:25:25.374 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:25:25.374 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:25:25.375 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:25:25.406 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:25:25.443 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:25:25.444 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:25:25.485 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:25:26.515 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:27:26.562 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:27:26.600 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:27:27.577 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:27:27.594 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:27:27.594 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:27:27.594 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:27:27.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:27:27.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:27:27.855 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:27:28.045 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:27:28.076 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:27:28.076 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:27:28.112 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:27:28.112 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:27:28.113 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:27:28.147 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:27:28.179 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:27:28.181 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:27:28.215 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:27:29.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:29:29.232 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:29:29.268 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:29:30.289 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:29:30.309 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:29:30.310 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:29:30.310 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:29:30.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:29:30.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:29:30.559 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:29:30.765 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:29:30.796 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:29:30.796 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:29:30.830 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:29:30.831 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:29:30.832 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:29:30.865 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:29:30.897 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:29:30.899 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:29:30.936 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:29:31.924 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:31:31.971 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:31:32.011 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:31:33.118 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:31:33.135 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:31:33.136 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:31:33.136 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:31:33.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:31:33.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:31:33.385 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:31:33.584 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:31:33.614 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:31:33.614 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:31:33.652 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:31:33.652 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:31:33.653 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:31:33.684 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:31:33.720 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:31:33.722 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:31:33.760 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:31:34.799 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:33:34.850 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:33:34.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:33:35.907 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:33:35.923 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:33:35.924 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:33:35.924 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:33:35.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:33:36.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:33:36.182 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:33:36.377 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:33:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:33:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:33:36.444 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:33:36.444 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:33:36.445 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:33:36.473 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:33:36.512 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:33:36.514 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:33:36.554 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:33:37.568 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:35:37.624 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:35:37.664 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:35:38.661 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:35:38.680 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:35:38.680 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:35:38.680 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:35:38.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:35:38.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:35:38.924 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:35:39.158 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:35:39.158 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:35:39.196 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:35:39.228 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:35:39.261 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:35:39.263 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:35:39.303 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:35:40.358 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:37:40.406 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:37:40.442 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:37:41.440 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:37:41.456 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:37:41.456 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:37:41.456 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:37:41.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:37:41.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:37:41.699 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:37:41.941 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:37:41.941 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:37:41.980 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:37:42.011 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:37:42.046 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:37:42.048 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:37:42.084 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:37:43.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:39:43.119 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:39:43.154 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:39:44.133 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:39:44.150 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:39:44.150 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:39:44.150 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:39:44.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:39:44.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:39:44.406 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:39:44.633 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:39:44.634 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:39:44.673 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:39:44.703 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:39:44.740 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:39:44.741 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:39:44.776 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:39:45.782 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:41:45.832 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:41:45.868 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:41:46.907 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:41:46.926 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:41:46.926 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:41:46.926 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:41:46.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:41:47.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:41:47.167 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:41:47.401 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:41:47.401 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:41:47.441 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:41:47.470 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:41:47.506 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:41:47.507 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:41:47.545 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:41:48.566 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:43:48.620 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:43:48.667 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:43:49.679 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:43:49.694 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:43:49.695 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:43:49.695 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:43:49.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:43:49.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:43:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:43:50.186 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:43:50.186 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:43:50.226 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:43:50.257 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:43:50.294 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:43:50.296 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:43:50.331 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:43:51.290 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:45:51.340 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:45:51.373 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:45:52.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:45:52.299 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:45:52.299 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:45:52.299 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:45:52.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:45:52.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:45:52.556 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:45:52.786 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:45:52.786 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:45:52.827 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:45:52.855 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:45:52.894 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:45:52.895 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:45:52.928 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:45:53.867 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:47:53.915 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:47:53.954 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:47:54.969 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:47:54.987 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:47:54.987 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:47:54.987 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:47:54.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:47:55.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:47:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:47:55.499 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:47:55.499 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:47:55.531 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:47:55.568 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:47:55.598 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:47:55.599 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:47:55.707 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:47:56.714 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:49:56.763 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:49:56.801 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:49:57.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:49:57.833 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:49:57.833 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:49:57.833 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:49:57.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:49:58.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:49:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:49:58.326 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:49:58.326 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:49:58.367 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:49:58.396 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:49:58.434 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:49:58.435 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:49:58.472 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:49:59.463 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:51:59.512 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:51:59.552 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:52:00.645 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:52:00.662 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:52:00.663 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:52:00.663 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:52:00.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:52:00.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:52:00.924 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:52:01.148 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:52:01.149 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:52:01.190 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:52:01.217 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:52:01.257 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:52:01.259 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:52:01.294 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:52:02.309 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:54:02.358 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:54:02.397 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:54:03.437 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:54:03.454 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:54:03.455 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:54:03.455 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:54:03.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:54:03.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:54:03.703 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:54:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:54:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:54:03.999 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:54:04.036 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:54:04.068 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:54:04.069 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:54:04.107 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:54:05.130 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:56:05.187 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:56:05.229 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:56:06.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:56:06.257 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:56:06.258 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:56:06.258 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:56:06.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:56:06.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:56:06.508 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:56:06.755 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:56:06.755 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:56:06.798 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:56:06.827 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:56:06.867 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:56:06.868 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:56:06.905 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:56:07.909 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:58:07.958 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T15:58:07.996 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:58:09.004 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T15:58:09.020 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T15:58:09.020 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T15:58:09.020 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:58:09.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:58:09.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T15:58:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T15:58:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T15:58:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T15:58:09.546 INFO:teuthology.misc:Compressing logs... 2021-03-20T15:58:09.574 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T15:58:09.613 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T15:58:09.614 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T15:58:09.650 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T15:58:10.637 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:00:10.689 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:00:10.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:00:11.849 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:00:11.866 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:00:11.866 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:00:11.866 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:00:11.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:00:12.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:00:12.111 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:00:12.371 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:00:12.371 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:00:12.405 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:00:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:00:12.471 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:00:12.473 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:00:12.512 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:00:13.610 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:02:13.666 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:02:13.701 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:02:14.738 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:02:14.753 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:02:14.753 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:02:14.754 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:02:14.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:02:14.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:02:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:02:15.241 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:02:15.241 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:02:15.281 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:02:15.311 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:02:15.348 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:02:15.349 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:02:15.386 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:02:16.420 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:04:16.472 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:04:16.509 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:04:17.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:04:17.520 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:04:17.520 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:04:17.520 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:04:17.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:04:17.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:04:17.779 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:04:18.004 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:04:18.004 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:04:18.045 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:04:18.074 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:04:18.111 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:04:18.112 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:04:18.147 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:04:19.131 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:06:19.178 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:06:19.213 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:06:20.205 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:06:20.235 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:06:20.236 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:06:20.236 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:06:20.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:06:20.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:06:20.528 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:06:20.792 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:06:20.792 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:06:20.822 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:06:20.862 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:06:20.888 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:06:20.890 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:06:20.924 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:06:21.880 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:08:21.925 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:08:21.962 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:08:23.042 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:08:23.062 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:08:23.063 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:08:23.063 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:08:23.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:08:23.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:08:23.305 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:08:23.557 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:08:23.557 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:08:23.597 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:08:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:08:23.663 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:08:23.665 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:08:23.703 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:08:24.725 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:10:24.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:10:24.813 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:10:25.821 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:10:25.838 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:10:25.838 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:10:25.838 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:10:25.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:10:26.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:10:26.100 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:10:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:10:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:10:26.368 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:10:26.395 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:10:26.436 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:10:26.437 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:10:26.474 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:10:27.532 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:12:27.579 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:12:27.617 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:12:28.667 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:12:28.683 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:12:28.683 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:12:28.683 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:12:28.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:12:28.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:12:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:12:29.200 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:12:29.201 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:12:29.237 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:12:29.273 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:12:29.312 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:12:29.314 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:12:29.351 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:12:30.356 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:14:30.408 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:14:30.444 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:14:31.538 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:14:31.555 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:14:31.555 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:14:31.555 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:14:31.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:14:31.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:14:31.800 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:14:32.052 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:14:32.052 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:14:32.097 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:14:32.125 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:14:32.165 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:14:32.171 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:14:32.207 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:14:33.264 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:16:33.316 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:16:33.354 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:16:34.364 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:16:34.382 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:16:34.382 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:16:34.382 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:16:34.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:16:34.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:16:34.640 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:16:34.870 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:16:34.870 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:16:34.911 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:16:34.939 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:16:34.978 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:16:34.979 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:16:35.018 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:16:36.049 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:18:36.106 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:18:36.142 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:18:37.152 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:18:37.168 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:18:37.168 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:18:37.169 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:18:37.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:18:37.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:18:37.410 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:18:37.669 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:18:37.670 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:18:37.700 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:18:37.739 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:18:37.766 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:18:37.767 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:18:37.803 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:18:38.763 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:20:38.816 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:20:38.850 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:20:39.856 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:20:39.872 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:20:39.873 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:20:39.873 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:20:39.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:20:40.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:20:40.115 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:20:40.358 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:20:40.358 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:20:40.399 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:20:40.427 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:20:40.467 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:20:40.468 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:20:40.505 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:20:41.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:22:41.591 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:22:41.685 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:22:43.528 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:22:43.546 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:22:43.546 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:22:43.546 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:22:43.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:22:43.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:22:43.805 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:22:44.033 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:22:44.034 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:22:44.075 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:22:44.102 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:22:44.143 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:22:44.144 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:22:44.182 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:22:45.972 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:24:46.021 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:24:46.058 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:24:47.758 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:24:47.773 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:24:47.773 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:24:47.773 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:24:47.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:24:47.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:24:48.017 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:24:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:24:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:24:48.307 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:24:48.348 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:24:48.374 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:24:48.375 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:24:48.413 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:24:49.401 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:26:49.451 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:26:49.488 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:26:50.446 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:26:50.462 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:26:50.462 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:26:50.463 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:26:50.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:26:50.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:26:50.707 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:26:50.954 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:26:50.954 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:26:50.997 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:26:51.024 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:26:51.067 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:26:51.068 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:26:51.101 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:26:52.074 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:28:52.122 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:28:52.162 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:28:53.164 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:28:53.180 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:28:53.180 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:28:53.180 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:28:53.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:28:53.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:28:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:28:53.667 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:28:53.667 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:28:53.712 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:28:53.739 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:28:53.779 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:28:53.781 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:28:53.818 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:28:54.867 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:30:54.925 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:30:54.959 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:30:55.952 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:30:55.968 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:30:55.968 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:30:55.969 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:30:55.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:30:56.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:30:56.222 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:30:56.484 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:30:56.484 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:30:56.515 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:30:56.554 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:30:56.582 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:30:56.584 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:30:56.623 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:30:57.646 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:32:57.693 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:32:57.731 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:32:58.746 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:32:58.764 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:32:58.764 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:32:58.765 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:32:58.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:32:58.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:32:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:32:59.261 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:32:59.262 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:32:59.303 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:32:59.331 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:32:59.370 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:32:59.372 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:32:59.410 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:33:00.357 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:35:00.404 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:35:00.446 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:35:01.446 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:35:01.463 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:35:01.463 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:35:01.463 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:35:01.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:35:01.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:35:01.727 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:35:01.950 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:35:01.950 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:35:01.990 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:35:02.021 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:35:02.056 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:35:02.058 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:35:02.093 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:35:03.097 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:37:03.146 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:37:03.185 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:37:04.254 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:37:04.271 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:37:04.271 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:37:04.271 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:37:04.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:37:04.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:37:04.515 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:37:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:37:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:37:04.805 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:37:04.843 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:37:04.872 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:37:04.873 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:37:04.913 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:37:05.986 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:39:06.036 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:39:06.072 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:39:07.144 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:39:07.161 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:39:07.161 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:39:07.161 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:39:07.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:39:07.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:39:07.402 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:39:07.647 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:39:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:39:07.691 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:39:07.719 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:39:07.759 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:39:07.761 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:39:07.797 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:39:08.805 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:41:08.851 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:41:08.889 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:41:09.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:41:09.886 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:41:09.886 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:41:09.886 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:41:09.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:41:10.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:41:10.148 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:41:10.375 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:41:10.375 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:41:10.416 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:41:10.444 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:41:10.486 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:41:10.488 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:41:10.524 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:41:11.507 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:43:11.557 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:43:11.592 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:43:12.557 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:43:12.573 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:43:12.573 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:43:12.573 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:43:12.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:43:12.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:43:12.814 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:43:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:43:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:43:13.105 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:43:13.144 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:43:13.172 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:43:13.174 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:43:13.208 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:43:14.179 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:45:14.225 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:45:14.259 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:45:15.264 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:45:15.280 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:45:15.280 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:45:15.281 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:45:15.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:45:15.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:45:15.523 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:45:15.770 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:45:15.770 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:45:15.810 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:45:15.838 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:45:15.877 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:45:15.878 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:45:15.913 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:45:16.874 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:47:16.921 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:47:16.956 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:47:17.931 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:47:17.947 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:47:17.947 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:47:17.947 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:47:17.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:47:18.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:47:18.206 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:47:18.432 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:47:18.433 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:47:18.473 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:47:18.501 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:47:18.541 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:47:18.543 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:47:18.576 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:47:19.580 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:49:19.629 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:49:19.665 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:49:20.636 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:49:20.653 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:49:20.653 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:49:20.654 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:49:20.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:49:20.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:49:20.895 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:49:21.155 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:49:21.155 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:49:21.186 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:49:21.225 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:49:21.252 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:49:21.253 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:49:21.288 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:49:22.223 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:51:22.269 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:51:22.306 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:51:23.337 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:51:23.356 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:51:23.356 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:51:23.357 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:51:23.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:51:23.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:51:23.603 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:51:23.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:51:23.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:51:23.897 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:51:23.927 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:51:23.966 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:51:23.966 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:51:24.003 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:51:24.988 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:53:25.035 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:53:25.074 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:53:26.053 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:53:26.070 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:53:26.070 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:53:26.071 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:53:26.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:53:26.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:53:26.333 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:53:26.566 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:53:26.566 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:53:26.606 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:53:26.636 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:53:26.673 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:53:26.674 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:53:26.708 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:53:27.713 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:55:27.769 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:55:27.802 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:55:28.743 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:55:28.760 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:55:28.760 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:55:28.760 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:55:28.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:55:28.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:55:29.000 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:55:29.242 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:55:29.242 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:55:29.281 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:55:29.312 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:55:29.347 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:55:29.348 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:55:29.426 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:55:30.444 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:57:30.492 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:57:30.526 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:57:31.577 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:57:31.594 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:57:31.594 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:57:31.594 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:57:31.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:57:31.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:57:31.838 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:57:32.083 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:57:32.083 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:57:32.122 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:57:32.154 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:57:32.189 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:57:32.191 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:57:32.229 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:57:33.192 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:59:33.239 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T16:59:33.273 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:59:34.212 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T16:59:34.227 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T16:59:34.227 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T16:59:34.227 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:59:34.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:59:34.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T16:59:34.486 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T16:59:34.716 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T16:59:34.716 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T16:59:34.758 INFO:teuthology.misc:Compressing logs... 2021-03-20T16:59:34.786 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T16:59:34.825 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T16:59:34.827 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T16:59:34.861 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T16:59:35.797 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:01:35.847 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:01:35.884 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:01:37.970 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:01:37.987 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:01:37.987 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:01:37.987 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:01:37.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:01:38.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:01:38.231 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:01:38.486 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:01:38.487 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:01:38.517 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:01:38.555 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:01:38.583 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:01:38.584 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:01:38.623 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:01:39.703 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:03:39.753 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:03:39.786 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:03:40.740 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:03:40.756 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:03:40.756 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:03:40.756 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:03:40.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:03:40.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:03:41.000 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:03:41.241 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:03:41.242 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:03:41.282 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:03:41.310 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:03:41.350 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:03:41.351 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:03:41.384 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:03:42.415 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:05:42.465 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:05:42.499 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:05:43.457 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:05:43.474 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:05:43.475 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:05:43.475 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:05:43.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:05:43.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:05:43.736 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:05:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:05:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:05:44.008 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:05:44.038 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:05:44.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:05:44.077 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:05:44.109 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:05:45.030 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:07:45.079 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:07:45.116 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:07:46.109 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:07:46.125 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:07:46.125 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:07:46.125 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:07:46.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:07:46.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:07:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:07:46.624 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:07:46.624 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:07:46.654 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:07:46.693 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:07:46.720 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:07:46.722 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:07:46.759 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:07:47.762 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:09:47.809 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:09:47.846 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:09:48.881 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:09:48.898 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:09:48.899 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:09:48.899 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:09:48.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:09:49.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:09:49.150 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:09:49.391 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:09:49.392 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:09:49.433 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:09:49.462 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:09:49.500 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:09:49.501 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:09:49.539 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:09:50.582 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:11:50.630 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:11:50.666 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:11:51.619 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:11:51.635 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:11:51.635 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:11:51.635 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:11:51.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:11:51.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:11:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:11:52.120 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:11:52.120 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:11:52.162 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:11:52.189 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:11:52.230 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:11:52.231 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:11:52.264 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:11:53.244 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:13:53.295 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:13:53.334 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:13:54.398 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:13:54.414 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:13:54.415 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:13:54.415 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:13:54.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:13:54.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:13:54.659 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:13:54.917 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:13:54.917 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:13:54.947 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:13:54.987 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:13:55.013 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:13:55.014 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:13:55.055 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:13:56.048 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:15:56.099 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:15:56.135 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:15:57.095 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:15:57.130 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:15:57.130 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:15:57.131 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:15:57.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:15:57.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:15:57.384 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:15:57.637 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:15:57.637 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:15:57.678 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:15:57.706 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:15:57.745 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:15:57.746 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:15:57.781 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:15:58.713 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:17:58.760 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:17:58.801 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:17:59.836 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:17:59.853 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:17:59.853 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:17:59.853 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:17:59.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:18:00.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:18:00.112 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:18:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:18:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:18:00.380 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:18:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:18:00.446 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:18:00.448 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:18:00.488 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:18:01.483 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:20:01.533 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:20:01.568 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:20:02.530 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:20:02.546 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:20:02.546 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:20:02.546 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:20:02.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:20:02.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:20:02.786 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:20:03.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:20:03.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:20:03.072 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:20:03.110 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:20:03.142 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:20:03.143 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:20:03.212 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:20:04.147 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:22:04.197 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:22:04.234 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:22:05.185 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:22:05.200 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:22:05.200 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:22:05.200 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:22:05.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:22:05.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:22:05.439 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:22:05.722 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:22:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:22:05.790 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:22:05.791 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:22:05.824 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:22:06.777 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:24:06.826 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:24:06.860 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:24:07.828 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:24:07.843 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:24:07.843 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:24:07.844 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:24:07.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:24:08.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:24:08.105 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:24:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:24:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:24:08.371 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:24:08.397 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:24:08.437 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:24:08.439 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:24:08.473 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:24:09.421 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:26:09.470 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:26:09.503 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:26:10.464 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:26:10.480 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:26:10.480 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:26:10.480 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:26:10.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:26:10.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:26:10.721 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:26:10.984 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:26:10.984 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:26:11.016 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:26:11.054 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:26:11.083 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:26:11.085 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:26:11.120 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:26:12.064 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:28:12.112 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:28:12.147 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:28:13.138 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:28:13.154 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:28:13.154 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:28:13.154 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:28:13.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:28:13.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:28:13.399 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:28:13.642 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:28:13.642 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:28:13.683 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:28:13.711 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:28:13.750 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:28:13.751 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:28:13.787 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:28:14.768 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:30:14.817 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:30:14.855 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:30:15.794 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:30:15.854 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:30:15.854 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:30:15.854 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:30:15.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:30:16.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:30:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:30:16.337 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:30:16.337 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:30:16.378 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:30:16.406 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:30:16.444 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:30:16.446 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:30:16.481 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:30:17.466 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:32:17.514 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:32:17.551 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:32:18.544 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:32:18.560 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:32:18.560 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:32:18.560 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:32:18.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:32:18.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:32:18.806 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:32:19.063 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:32:19.063 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:32:19.093 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:32:19.131 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:32:19.160 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:32:19.162 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:32:19.200 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:32:20.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:34:20.239 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:34:20.274 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:34:21.238 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:34:21.255 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:34:21.255 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:34:21.255 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:34:21.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:34:21.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:34:21.500 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:34:21.741 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:34:21.741 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:34:21.780 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:34:21.810 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:34:21.847 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:34:21.849 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:34:21.885 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:34:22.841 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:36:22.889 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:36:22.926 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:36:23.885 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:36:23.903 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:36:23.903 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:36:23.904 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:36:23.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:36:24.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:36:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:36:24.398 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:36:24.398 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:36:24.440 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:36:24.468 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:36:24.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:36:24.512 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:36:24.547 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:36:25.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:38:25.551 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:38:25.591 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:38:26.604 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:38:26.621 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:38:26.621 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:38:26.621 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:38:26.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:38:26.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:38:26.866 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:38:27.126 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:38:27.126 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:38:27.162 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:38:27.196 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:38:27.230 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:38:27.232 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:38:27.272 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:38:28.289 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:40:28.337 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:40:28.371 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:40:29.346 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:40:29.364 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:40:29.364 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:40:29.364 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:40:29.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:40:29.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:40:29.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:40:29.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:40:29.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:40:29.892 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:40:29.919 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:40:29.959 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:40:29.961 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:40:29.999 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:40:31.010 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:42:31.071 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:42:31.109 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:42:32.115 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:42:32.136 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:42:32.136 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:42:32.136 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:42:32.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:42:32.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:42:32.395 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:42:32.624 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:42:32.624 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:42:32.665 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:42:32.692 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:42:32.733 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:42:32.734 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:42:32.771 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:42:33.761 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:44:33.814 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:44:33.849 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:44:34.860 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:44:34.877 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:44:34.878 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:44:34.878 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:44:34.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:44:35.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:44:35.124 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:44:35.391 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:44:35.392 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:44:35.422 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:44:35.462 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:44:35.490 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:44:35.492 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:44:35.529 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:44:36.504 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:46:36.553 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:46:36.591 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:46:37.678 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:46:37.695 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:46:37.696 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:46:37.696 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:46:37.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:46:37.912 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:46:37.943 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:46:38.192 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:46:38.192 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:46:38.233 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:46:38.262 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:46:38.300 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:46:38.301 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:46:38.336 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:46:39.317 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:48:39.376 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:48:39.412 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:48:40.390 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:48:40.408 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:48:40.408 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:48:40.408 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:48:40.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:48:40.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:48:40.663 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:48:40.889 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:48:40.889 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:48:40.930 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:48:40.958 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:48:40.996 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:48:40.997 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:48:41.034 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:48:41.989 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:50:42.074 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:50:42.109 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:50:43.085 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:50:43.101 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:50:43.101 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:50:43.101 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:50:43.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:50:43.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:50:43.346 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:50:43.611 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:50:43.611 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:50:43.643 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:50:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:50:43.711 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:50:43.711 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:50:43.748 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:50:44.720 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:52:44.772 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:52:44.810 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:52:45.817 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:52:45.836 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:52:45.836 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:52:45.836 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:52:45.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:52:46.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:52:46.080 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:52:46.323 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:52:46.323 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:52:46.365 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:52:46.393 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:52:46.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:52:46.434 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:52:46.470 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:52:47.475 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:54:47.527 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:54:47.564 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:54:48.571 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:54:48.590 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:54:48.590 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:54:48.591 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:54:48.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:54:48.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:54:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:54:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:54:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:54:49.117 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:54:49.145 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:54:49.184 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:54:49.185 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:54:49.224 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:54:50.221 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:56:50.282 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:56:50.316 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:56:51.244 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:56:51.259 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:56:51.259 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:56:51.259 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:56:51.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:56:51.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:56:51.508 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:56:51.773 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:56:51.774 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:56:51.805 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:56:51.845 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:56:51.874 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:56:51.875 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:56:51.908 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:56:52.819 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:58:52.869 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T17:58:52.922 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:58:53.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T17:58:53.918 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T17:58:53.918 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T17:58:53.918 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:58:53.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:58:54.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T17:58:54.163 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T17:58:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T17:58:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T17:58:54.449 INFO:teuthology.misc:Compressing logs... 2021-03-20T17:58:54.477 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T17:58:54.518 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T17:58:54.519 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T17:58:54.555 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T17:58:55.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:00:55.570 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:00:55.607 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:00:56.616 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:00:56.635 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:00:56.635 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:00:56.635 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:00:56.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:00:56.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:00:57.007 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:00:57.292 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:00:57.292 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:00:57.333 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:00:57.363 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:00:57.400 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:00:57.401 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:00:57.438 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:00:58.499 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:02:58.545 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:02:58.583 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:02:59.617 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:02:59.633 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:02:59.634 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:02:59.634 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:02:59.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:02:59.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:02:59.878 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:03:00.141 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:03:00.141 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:03:00.172 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:03:00.212 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:03:00.239 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:03:00.241 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:03:00.280 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:03:01.300 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:05:01.352 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:05:01.446 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:05:03.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:05:03.354 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:05:03.354 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:05:03.355 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:05:03.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:05:03.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:05:03.598 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:05:03.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:05:03.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:05:03.885 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:05:03.912 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:05:03.952 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:05:03.954 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:05:03.999 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:05:05.252 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:07:05.297 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:07:05.337 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:07:07.277 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:07:07.295 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:07:07.295 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:07:07.295 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:07:07.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:07:07.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:07:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:07:07.809 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:07:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:07:07.862 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:07:07.891 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:07:07.939 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:07:07.941 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:07:08.077 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:07:10.527 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:09:10.578 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:09:10.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:09:12.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:09:12.510 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:09:12.510 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:09:12.510 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:09:12.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:09:12.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:09:12.756 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:09:13.014 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:09:13.014 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:09:13.044 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:09:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:09:13.110 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:09:13.112 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:09:13.153 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:09:15.030 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:11:15.078 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:11:15.131 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:11:16.974 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:11:16.995 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:11:16.995 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:11:16.995 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:11:16.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:11:17.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:11:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:11:17.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:11:17.492 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:11:17.532 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:11:17.562 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:11:17.601 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:11:17.603 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:11:17.644 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:11:18.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:13:18.888 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:13:18.927 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:13:20.708 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:13:20.728 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:13:20.728 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:13:20.728 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:13:20.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:13:20.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:13:20.991 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:13:21.218 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:13:21.219 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:13:21.263 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:13:21.290 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:13:21.330 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:13:21.332 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:13:21.374 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:13:22.459 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:15:22.509 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:15:22.543 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:15:24.553 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:15:24.570 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:15:24.571 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:15:24.571 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:15:24.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:15:24.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:15:24.816 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:15:25.080 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:15:25.080 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:15:25.114 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:15:25.150 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:15:25.180 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:15:25.182 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:15:25.217 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:15:26.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:17:26.895 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:17:26.935 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:17:28.717 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:17:28.735 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:17:28.735 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:17:28.736 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:17:28.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:17:28.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:17:28.980 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:17:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:17:29.226 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:17:29.268 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:17:29.295 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:17:29.335 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:17:29.336 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:17:29.374 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:17:31.352 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:19:31.403 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:19:31.442 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:19:33.217 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:19:33.235 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:19:33.235 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:19:33.235 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:19:33.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:19:33.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:19:33.496 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:19:33.720 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:19:33.721 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:19:33.761 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:19:33.788 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:19:33.832 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:19:33.832 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:19:33.872 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:19:35.625 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:21:35.676 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:21:35.712 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:21:37.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:21:37.559 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:21:37.559 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:21:37.560 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:21:37.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:21:37.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:21:37.803 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:21:38.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:21:38.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:21:38.090 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:21:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:21:38.156 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:21:38.158 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:21:38.194 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:21:40.046 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:23:40.099 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:23:40.136 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:23:41.891 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:23:41.908 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:23:41.908 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:23:41.908 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:23:41.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:23:42.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:23:42.153 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:23:42.402 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:23:42.402 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:23:42.441 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:23:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:23:42.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:23:42.510 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:23:42.546 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:23:43.579 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:25:43.628 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:25:43.663 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:25:45.511 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:25:45.528 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:25:45.528 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:25:45.528 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:25:45.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:25:45.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:25:45.791 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:25:46.016 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:25:46.017 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:25:46.059 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:25:46.087 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:25:46.126 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:25:46.127 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:25:46.166 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:25:47.374 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:27:47.425 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:27:47.463 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:27:49.303 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:27:49.321 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:27:49.321 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:27:49.321 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:27:49.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:27:49.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:27:49.563 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:27:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:27:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:27:49.862 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:27:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:27:49.930 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:27:49.931 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:27:49.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:27:51.511 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:29:51.561 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:29:51.598 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:29:53.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:29:53.278 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:29:53.278 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:29:53.278 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:29:53.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:29:53.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:29:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:29:53.770 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:29:53.770 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:29:53.811 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:29:53.837 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:29:53.878 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:29:53.879 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:29:53.917 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:29:54.970 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:31:55.020 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:31:55.057 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:31:56.134 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:31:56.152 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:31:56.152 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:31:56.152 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:31:56.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:31:56.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:31:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:31:56.651 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:31:56.652 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:31:56.694 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:31:56.721 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:31:56.762 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:31:56.763 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:31:56.800 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:31:57.848 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:33:57.897 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:33:57.935 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:33:58.958 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:33:58.975 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:33:58.975 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:33:58.975 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:33:58.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:33:59.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:33:59.220 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:33:59.487 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:33:59.487 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:33:59.520 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:33:59.558 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:33:59.588 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:33:59.590 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:33:59.627 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:34:00.671 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:36:00.721 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:36:00.762 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:36:01.841 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:36:01.857 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:36:01.857 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:36:01.857 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:36:01.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:36:02.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:36:02.103 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:36:02.345 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:36:02.345 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:36:02.386 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:36:02.413 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:36:02.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:36:02.454 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:36:02.492 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:36:03.514 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:38:03.567 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:38:03.611 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:38:04.656 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:38:04.674 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:38:04.674 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:38:04.674 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:38:04.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:38:04.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:38:04.936 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:38:05.170 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:38:05.170 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:38:05.213 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:38:05.241 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:38:05.281 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:38:05.283 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:38:05.364 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:38:06.377 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:40:06.430 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:40:06.468 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:40:07.487 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:40:07.503 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:40:07.503 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:40:07.503 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:40:07.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:40:07.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:40:07.746 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:40:08.004 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:40:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:40:08.035 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:40:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:40:08.102 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:40:08.103 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:40:08.140 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:40:09.203 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:42:09.257 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:42:09.295 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:42:10.281 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:42:10.297 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:42:10.298 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:42:10.298 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:42:10.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:42:10.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:42:10.542 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:42:10.788 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:42:10.788 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:42:10.827 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:42:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:42:10.895 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:42:10.897 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:42:10.935 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:42:11.908 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:44:11.957 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:44:11.995 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:44:13.002 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:44:13.018 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:44:13.019 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:44:13.019 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:44:13.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:44:13.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:44:13.278 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:44:13.505 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:44:13.506 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:44:13.547 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:44:13.574 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:44:13.614 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:44:13.615 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:44:13.654 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:44:14.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:46:14.690 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:46:14.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:46:15.666 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:46:15.682 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:46:15.682 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:46:15.683 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:46:15.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:46:15.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:46:15.927 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:46:16.189 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:46:16.189 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:46:16.218 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:46:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:46:16.284 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:46:16.286 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:46:16.321 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:46:17.278 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:48:17.332 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:48:17.382 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:48:18.448 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:48:18.466 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:48:18.466 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:48:18.466 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:48:18.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:48:18.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:48:18.719 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:48:18.967 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:48:18.967 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:48:19.006 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:48:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:48:19.074 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:48:19.077 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:48:19.113 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:48:20.161 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:50:20.212 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:50:20.252 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:50:21.278 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:50:21.294 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:50:21.295 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:50:21.295 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:50:21.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:50:21.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:50:21.555 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:50:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:50:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:50:21.826 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:50:21.852 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:50:21.893 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:50:21.895 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:50:21.931 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:50:22.965 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:52:23.022 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:52:23.061 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:52:24.134 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:52:24.152 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:52:24.152 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:52:24.153 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:52:24.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:52:24.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:52:24.399 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:52:24.656 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:52:24.656 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:52:24.685 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:52:24.724 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:52:24.752 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:52:24.753 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:52:24.792 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:52:25.883 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:54:25.930 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:54:25.975 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:54:27.061 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:54:27.078 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:54:27.079 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:54:27.079 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:54:27.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:54:27.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:54:27.324 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:54:27.567 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:54:27.567 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:54:27.607 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:54:27.636 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:54:27.674 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:54:27.675 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:54:27.711 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:54:28.714 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:56:28.772 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:56:28.811 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:56:29.884 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:56:29.905 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:56:29.905 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:56:29.905 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:56:29.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:56:30.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:56:30.172 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:56:30.406 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:56:30.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:56:30.449 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:56:30.476 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:56:30.519 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:56:30.520 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:56:30.558 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:56:31.591 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:58:31.712 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T18:58:31.750 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:58:32.848 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T18:58:32.865 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T18:58:32.865 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T18:58:32.866 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:58:32.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:58:33.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T18:58:33.111 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T18:58:33.376 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T18:58:33.377 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T18:58:33.413 INFO:teuthology.misc:Compressing logs... 2021-03-20T18:58:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T18:58:33.479 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T18:58:33.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T18:58:33.519 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T18:58:34.526 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:00:34.579 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:00:34.622 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:00:35.793 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:00:35.829 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:00:35.829 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:00:35.830 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:00:35.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:00:36.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:00:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:00:36.324 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:00:36.324 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:00:36.364 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:00:36.393 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:00:36.431 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:00:36.433 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:00:36.480 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:00:37.577 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:02:37.628 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:02:37.669 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:02:38.745 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:02:38.762 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:02:38.763 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:02:38.763 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:02:38.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:02:38.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:02:39.024 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:02:39.254 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:02:39.254 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:02:39.297 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:02:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:02:39.364 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:02:39.366 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:02:39.407 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:02:40.449 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:04:40.500 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:04:40.538 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:04:41.594 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:04:41.611 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:04:41.611 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:04:41.612 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:04:41.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:04:41.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:04:41.861 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:04:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:04:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:04:42.157 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:04:42.196 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:04:42.228 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:04:42.228 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:04:42.264 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:04:43.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:06:43.404 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:06:43.441 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:06:44.398 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:06:44.414 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:06:44.414 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:06:44.414 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:06:44.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:06:44.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:06:44.661 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:06:44.905 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:06:44.905 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:06:44.945 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:06:44.975 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:06:45.014 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:06:45.015 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:06:45.050 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:06:46.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:08:46.073 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:08:46.109 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:08:47.229 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:08:47.246 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:08:47.246 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:08:47.247 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:08:47.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:08:47.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:08:47.505 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:08:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:08:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:08:47.776 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:08:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:08:47.843 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:08:47.845 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:08:47.888 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:08:48.989 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:10:49.036 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:10:49.075 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:10:50.105 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:10:50.122 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:10:50.122 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:10:50.122 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:10:50.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:10:50.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:10:50.367 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:10:50.633 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:10:50.633 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:10:50.667 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:10:50.703 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:10:50.735 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:10:50.737 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:10:50.774 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:10:51.829 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:12:51.876 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:12:51.954 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:12:53.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:12:53.040 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:12:53.040 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:12:53.040 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:12:53.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:12:53.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:12:53.282 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:12:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:12:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:12:53.567 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:12:53.593 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:12:53.633 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:12:53.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:12:53.668 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:12:54.637 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:14:54.687 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:14:54.722 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:14:55.759 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:14:55.777 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:14:55.777 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:14:55.777 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:14:55.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:14:55.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:14:56.036 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:14:56.267 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:14:56.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:14:56.310 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:14:56.337 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:14:56.377 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:14:56.379 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:14:56.415 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:14:57.437 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:16:57.485 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:16:57.527 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:16:58.619 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:16:58.638 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:16:58.638 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:16:58.638 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:16:58.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:16:58.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:16:58.891 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:16:59.151 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:16:59.152 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:16:59.183 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:16:59.223 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:16:59.250 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:16:59.252 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:16:59.291 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:17:00.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:19:00.406 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:19:00.445 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:19:01.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:19:01.511 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:19:01.511 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:19:01.511 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:19:01.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:19:01.732 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:19:01.761 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:19:02.009 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:19:02.009 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:19:02.050 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:19:02.080 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:19:02.117 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:19:02.119 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:19:02.156 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:19:03.187 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:21:03.255 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:21:03.298 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:21:05.436 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:21:05.453 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:21:05.454 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:21:05.454 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:21:05.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:21:05.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:21:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:21:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:21:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:21:05.990 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:21:06.021 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:21:06.057 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:21:06.059 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:21:06.106 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:21:08.045 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:23:08.106 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:23:08.148 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:23:09.284 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:23:09.302 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:23:09.302 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:23:09.302 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:23:09.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:23:09.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:23:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:23:09.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:23:09.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:23:09.841 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:23:09.878 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:23:09.909 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:23:09.911 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:23:09.955 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:23:11.076 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:25:11.126 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:25:11.165 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:25:13.235 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:25:13.257 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:25:13.257 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:25:13.257 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:25:13.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:25:13.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:25:13.504 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:25:13.745 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:25:13.745 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:25:13.786 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:25:13.814 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:25:13.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:25:13.855 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:25:13.895 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:25:15.336 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:27:15.388 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:27:15.426 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:27:17.678 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:27:17.695 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:27:17.696 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:27:17.696 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:27:17.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:27:17.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:27:17.952 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:27:18.186 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:27:18.186 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:27:18.228 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:27:18.256 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:27:18.296 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:27:18.297 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:27:18.339 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:27:20.292 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:29:20.342 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:29:20.516 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:29:22.358 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:29:22.376 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:29:22.376 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:29:22.377 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:29:22.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:29:22.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:29:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:29:22.888 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:29:22.888 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:29:22.922 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:29:22.959 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:29:22.999 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:29:22.999 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:29:23.038 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:29:24.487 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:31:24.559 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:31:24.599 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:31:26.492 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:31:26.509 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:31:26.509 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:31:26.509 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:31:26.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:31:26.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:31:26.752 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:31:26.999 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:31:27.000 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:31:27.039 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:31:27.070 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:31:27.106 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:31:27.107 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:31:27.146 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:31:28.327 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:33:28.392 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:33:28.455 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:33:30.475 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:33:30.493 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:33:30.493 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:33:30.494 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:33:30.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:33:30.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:33:30.756 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:33:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:33:30.982 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:33:31.022 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:33:31.051 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:33:31.088 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:33:31.090 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:33:31.136 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:33:32.961 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:35:33.010 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:35:33.049 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:35:34.875 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:35:34.909 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:35:34.910 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:35:34.910 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:35:34.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:35:35.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:35:35.157 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:35:35.415 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:35:35.415 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:35:35.445 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:35:35.484 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:35:35.512 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:35:35.513 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:35:35.550 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:35:36.668 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:37:36.736 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:37:36.785 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:37:38.666 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:37:38.684 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:37:38.684 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:37:38.684 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:37:38.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:37:38.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:37:38.931 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:37:39.178 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:37:39.178 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:37:39.220 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:37:39.250 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:37:39.288 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:37:39.290 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:37:39.337 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:37:41.070 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:39:41.128 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:39:41.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:39:43.410 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:39:43.427 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:39:43.428 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:39:43.428 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:39:43.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:39:43.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:39:43.725 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:39:44.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:39:44.031 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:39:44.073 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:39:44.111 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:39:44.147 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:39:44.148 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:39:44.190 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:39:45.490 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:41:45.538 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:41:45.577 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:41:47.581 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:41:47.599 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:41:47.599 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:41:47.599 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:41:47.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:41:47.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:41:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:41:48.103 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:41:48.104 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:41:48.133 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:41:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:41:48.200 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:41:48.201 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:41:48.239 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:41:49.744 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:43:49.798 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:43:49.837 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:43:51.761 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:43:51.792 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:43:51.793 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:43:51.793 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:43:51.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:43:52.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:43:52.051 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:43:52.296 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:43:52.296 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:43:52.336 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:43:52.365 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:43:52.403 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:43:52.405 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:43:52.449 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:43:54.214 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:45:54.265 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:45:54.304 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:45:56.077 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:45:56.096 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:45:56.096 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:45:56.097 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:45:56.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:45:56.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:45:56.363 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:45:56.596 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:45:56.596 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:45:56.640 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:45:56.668 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:45:56.708 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:45:56.710 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:45:56.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:45:57.858 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:47:57.910 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:47:57.948 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:47:59.808 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:47:59.826 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:47:59.826 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:47:59.826 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:47:59.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:48:00.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:48:00.072 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:48:00.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:48:00.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:48:00.357 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:48:00.396 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:48:00.423 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:48:00.425 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:48:00.463 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:48:01.552 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:50:01.604 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:50:01.643 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:50:03.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:50:03.560 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:50:03.560 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:50:03.560 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:50:03.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:50:03.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:50:03.808 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:50:04.054 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:50:04.055 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:50:04.097 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:50:04.124 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:50:04.164 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:50:04.165 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:50:04.206 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:50:05.565 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:52:05.631 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:52:05.675 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:52:07.682 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:52:07.700 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:52:07.700 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:52:07.700 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:52:07.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:52:07.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:52:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:52:08.293 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:52:08.293 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:52:08.335 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:52:08.363 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:52:08.403 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:52:08.404 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:52:08.446 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:52:10.358 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:54:10.423 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:54:10.463 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:54:11.511 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:54:11.533 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:54:11.534 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:54:11.534 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:54:11.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:54:11.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:54:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:54:12.045 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:54:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:54:12.079 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:54:12.116 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:54:12.146 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:54:12.149 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:54:12.187 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:54:13.196 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:56:13.245 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:56:13.284 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:56:15.293 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:56:15.313 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:56:15.314 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:56:15.314 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:56:15.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:56:15.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:56:15.560 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:56:15.806 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:56:15.806 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:56:15.848 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:56:15.877 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:56:15.916 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:56:15.917 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:56:15.956 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:56:17.817 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:58:17.864 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T19:58:17.901 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:58:18.853 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T19:58:18.869 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T19:58:18.869 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T19:58:18.869 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:58:18.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:58:19.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T19:58:19.129 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T19:58:19.363 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T19:58:19.363 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T19:58:19.404 INFO:teuthology.misc:Compressing logs... 2021-03-20T19:58:19.432 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T19:58:19.472 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T19:58:19.473 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T19:58:19.510 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T19:58:20.542 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:00:20.592 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:00:20.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:00:21.788 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:00:21.808 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:00:21.809 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:00:21.809 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:00:21.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:00:22.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:00:22.055 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:00:22.314 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:00:22.315 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:00:22.348 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:00:22.384 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:00:22.416 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:00:22.420 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:00:22.467 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:00:23.646 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:02:23.698 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:02:23.738 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:02:24.805 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:02:24.825 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:02:24.825 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:02:24.825 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:02:24.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:02:25.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:02:25.072 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:02:25.318 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:02:25.318 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:02:25.358 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:02:25.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:02:25.424 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:02:25.426 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:02:25.463 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:02:26.537 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:04:26.591 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:04:26.630 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:04:27.708 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:04:27.726 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:04:27.727 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:04:27.727 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:04:27.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:04:27.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:04:27.989 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:04:28.219 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:04:28.219 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:04:28.260 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:04:28.289 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:04:28.327 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:04:28.328 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:04:28.368 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:04:29.424 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:06:29.471 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:06:29.508 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:06:30.573 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:06:30.589 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:06:30.589 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:06:30.590 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:06:30.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:06:30.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:06:30.834 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:06:31.094 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:06:31.095 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:06:31.126 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:06:31.165 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:06:31.193 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:06:31.195 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:06:31.232 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:06:32.272 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:08:32.324 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:08:32.363 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:08:33.412 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:08:33.444 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:08:33.445 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:08:33.445 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:08:33.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:08:33.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:08:33.692 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:08:33.945 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:08:33.945 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:08:33.986 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:08:34.016 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:08:34.053 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:08:34.055 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:08:34.124 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:08:35.173 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:10:35.227 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:10:35.267 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:10:36.386 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:10:36.405 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:10:36.405 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:10:36.405 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:10:36.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:10:36.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:10:36.672 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:10:36.897 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:10:36.897 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:10:36.939 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:10:36.969 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:10:37.007 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:10:37.009 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:10:37.047 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:10:38.130 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:12:38.189 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:12:38.227 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:12:39.304 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:12:39.321 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:12:39.322 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:12:39.322 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:12:39.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:12:39.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:12:39.568 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:12:39.826 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:12:39.826 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:12:39.856 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:12:39.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:12:39.923 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:12:39.924 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:12:39.961 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:12:41.005 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:14:41.054 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:14:41.095 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:14:42.161 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:14:42.181 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:14:42.181 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:14:42.181 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:14:42.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:14:42.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:14:42.431 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:14:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:14:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:14:42.716 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:14:42.746 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:14:42.783 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:14:42.785 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:14:42.826 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:14:43.881 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:16:43.932 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:16:43.971 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:16:44.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:16:45.009 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:16:45.009 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:16:45.009 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:16:45.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:16:45.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:16:45.275 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:16:45.550 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:16:45.550 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:16:45.587 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:16:45.622 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:16:45.654 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:16:45.656 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:16:45.694 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:16:46.715 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:18:46.762 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:18:46.801 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:18:47.890 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:18:47.907 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:18:47.907 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:18:47.907 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:18:47.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:18:48.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:18:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:18:48.413 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:18:48.413 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:18:48.446 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:18:48.483 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:18:48.511 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:18:48.513 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:18:48.551 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:18:49.618 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:20:49.682 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:20:49.719 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:20:50.800 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:20:50.956 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:20:50.957 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:20:50.957 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:20:50.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:20:51.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:20:51.216 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:20:51.462 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:20:51.463 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:20:51.503 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:20:51.532 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:20:51.570 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:20:51.571 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:20:51.660 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:20:52.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:22:52.770 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:22:52.807 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:22:53.832 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:22:53.848 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:22:53.848 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:22:53.848 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:22:53.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:22:54.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:22:54.109 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:22:54.333 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:22:54.334 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:22:54.377 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:22:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:22:54.443 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:22:54.445 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:22:54.481 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:22:55.507 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:24:55.561 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:24:55.606 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:24:56.658 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:24:56.675 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:24:56.676 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:24:56.676 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:24:56.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:24:56.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:24:56.929 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:24:57.196 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:24:57.196 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:24:57.227 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:24:57.266 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:24:57.294 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:24:57.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:24:57.336 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:24:58.417 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:26:58.470 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:26:58.514 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:27:00.276 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:27:00.293 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:27:00.293 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:27:00.293 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:27:00.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:27:00.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:27:00.536 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:27:00.778 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:27:00.778 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:27:00.820 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:27:00.847 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:27:00.887 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:27:00.888 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:27:00.928 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:27:02.015 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:29:02.073 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:29:02.111 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:29:03.218 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:29:03.245 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:29:03.246 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:29:03.246 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:29:03.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:29:03.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:29:03.522 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:29:03.764 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:29:03.765 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:29:03.806 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:29:03.837 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:29:03.875 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:29:03.876 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:29:03.932 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:29:05.012 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:31:05.062 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:31:05.102 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:31:06.146 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:31:06.163 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:31:06.163 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:31:06.163 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:31:06.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:31:06.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:31:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:31:06.667 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:31:06.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:31:06.698 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:31:06.738 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:31:06.765 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:31:06.766 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:31:06.805 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:31:07.844 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:33:07.920 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:33:07.970 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:33:08.974 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:33:08.991 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:33:08.991 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:33:08.991 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:33:08.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:33:09.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:33:09.236 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:33:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:33:09.480 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:33:09.520 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:33:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:33:09.588 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:33:09.589 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:33:09.626 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:33:10.614 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:35:10.667 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:35:10.708 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:35:11.737 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:35:11.753 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:35:11.753 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:35:11.753 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:35:11.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:35:11.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:35:12.016 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:35:12.248 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:35:12.249 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:35:12.289 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:35:12.319 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:35:12.356 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:35:12.358 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:35:12.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:35:13.408 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:37:13.458 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:37:13.497 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:37:14.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:37:14.521 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:37:14.521 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:37:14.521 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:37:14.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:37:14.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:37:14.764 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:37:15.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:37:15.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:37:15.064 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:37:15.101 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:37:15.131 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:37:15.133 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:37:15.171 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:37:16.179 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:39:16.233 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:39:16.272 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:39:17.330 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:39:17.349 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:39:17.349 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:39:17.349 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:39:17.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:39:17.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:39:17.595 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:39:17.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:39:17.844 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:39:17.884 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:39:17.912 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:39:17.951 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:39:17.952 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:39:17.987 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:39:18.981 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:41:19.027 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:41:19.072 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:41:20.142 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:41:20.159 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:41:20.159 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:41:20.159 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:41:20.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:41:20.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:41:20.417 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:41:20.648 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:41:20.648 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:41:20.690 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:41:20.716 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:41:20.757 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:41:20.759 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:41:20.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:41:21.885 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:43:21.947 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:43:21.987 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:43:23.025 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:43:23.042 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:43:23.042 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:43:23.043 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:43:23.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:43:23.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:43:23.287 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:43:23.542 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:43:23.543 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:43:23.572 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:43:23.610 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:43:23.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:43:23.640 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:43:23.678 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:43:24.712 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:45:24.763 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:45:24.801 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:45:25.818 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:45:25.839 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:45:25.839 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:45:25.839 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:45:25.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:45:26.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:45:26.083 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:45:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:45:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:45:26.370 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:45:26.398 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:45:26.437 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:45:26.438 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:45:26.475 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:45:27.573 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:47:27.626 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:47:27.663 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:47:28.714 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:47:28.734 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:47:28.734 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:47:28.734 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:47:28.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:47:28.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:47:29.001 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:47:29.230 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:47:29.231 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:47:29.273 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:47:29.301 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:47:29.340 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:47:29.341 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:47:29.379 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:47:30.426 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:49:30.476 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:49:30.535 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:49:31.536 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:49:31.554 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:49:31.554 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:49:31.554 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:49:31.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:49:31.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:49:31.797 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:49:32.054 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:49:32.055 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:49:32.085 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:49:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:49:32.151 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:49:32.153 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:49:32.189 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:49:33.183 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:51:33.230 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:51:33.268 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:51:34.252 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:51:34.268 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:51:34.268 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:51:34.268 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:51:34.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:51:34.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:51:34.511 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:51:34.758 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:51:34.759 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:51:34.799 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:51:34.827 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:51:34.865 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:51:34.867 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:51:34.901 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:51:35.925 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:53:35.973 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:53:36.009 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:53:36.979 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:53:37.003 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:53:37.003 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:53:37.003 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:53:37.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:53:37.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:53:37.269 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:53:37.498 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:53:37.498 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:53:37.541 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:53:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:53:37.608 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:53:37.609 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:53:37.644 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:53:38.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:55:38.690 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:55:38.771 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:55:39.786 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:55:39.802 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:55:39.802 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:55:39.802 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:55:39.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:55:40.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:55:40.045 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:55:40.307 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:55:40.307 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:55:40.341 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:55:40.378 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:55:40.408 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:55:40.410 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:55:40.447 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:55:41.433 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:57:41.483 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:57:41.520 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:57:42.507 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:57:42.526 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:57:42.527 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:57:42.527 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:57:42.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:57:42.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:57:42.771 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:57:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:57:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:57:43.056 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:57:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:57:43.124 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:57:43.126 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:57:43.161 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:57:44.151 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:59:44.201 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T20:59:44.238 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:59:45.246 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T20:59:45.263 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T20:59:45.263 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T20:59:45.264 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:59:45.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:59:45.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T20:59:45.523 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T20:59:45.752 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T20:59:45.752 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T20:59:45.794 INFO:teuthology.misc:Compressing logs... 2021-03-20T20:59:45.822 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T20:59:45.862 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T20:59:45.864 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T20:59:45.899 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T20:59:46.896 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:01:46.943 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:01:46.982 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:01:48.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:01:48.061 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:01:48.061 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:01:48.061 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:01:48.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:01:48.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:01:48.303 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:01:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:01:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:01:48.598 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:01:48.635 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:01:48.664 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:01:48.666 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:01:48.704 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:01:49.754 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:03:49.811 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:03:49.849 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:03:50.876 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:03:50.893 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:03:50.894 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:03:50.894 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:03:50.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:03:51.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:03:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:03:51.379 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:03:51.379 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:03:51.418 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:03:51.448 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:03:51.485 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:03:51.486 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:03:51.524 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:03:52.545 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:05:52.594 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:05:52.630 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:05:53.653 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:05:53.669 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:05:53.669 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:05:53.669 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:05:53.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:05:53.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:05:53.928 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:05:54.152 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:05:54.152 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:05:54.193 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:05:54.220 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:05:54.259 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:05:54.261 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:05:54.297 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:05:55.255 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:07:55.307 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:07:55.345 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:07:56.370 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:07:56.386 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:07:56.386 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:07:56.386 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:07:56.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:07:56.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:07:56.637 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:07:56.900 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:07:56.900 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:07:56.931 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:07:56.970 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:07:56.997 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:07:57.000 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:07:57.037 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:07:58.018 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:09:58.068 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:09:58.101 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:09:59.094 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:09:59.110 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:09:59.110 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:09:59.110 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:09:59.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:09:59.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:09:59.353 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:09:59.604 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:09:59.604 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:09:59.645 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:09:59.674 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:09:59.711 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:09:59.712 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:09:59.749 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:10:00.712 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:12:00.759 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:12:00.793 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:12:01.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:12:01.884 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:12:01.884 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:12:01.884 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:12:01.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:12:02.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:12:02.146 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:12:02.381 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:12:02.381 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:12:02.422 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:12:02.450 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:12:02.489 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:12:02.491 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:12:02.525 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:12:03.464 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:14:03.513 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:14:03.549 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:14:04.560 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:14:04.576 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:14:04.576 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:14:04.576 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:14:04.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:14:04.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:14:04.820 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:14:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:14:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:14:05.106 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:14:05.145 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:14:05.172 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:14:05.173 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:14:05.209 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:14:06.210 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:16:06.264 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:16:06.301 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:16:07.388 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:16:07.406 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:16:07.407 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:16:07.407 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:16:07.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:16:07.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:16:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:16:07.896 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:16:07.896 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:16:07.935 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:16:07.965 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:16:08.001 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:16:08.003 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:16:08.041 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:16:09.067 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:18:09.115 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:18:09.151 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:18:10.189 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:18:10.205 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:18:10.205 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:18:10.206 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:18:10.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:18:10.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:18:10.467 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:18:10.697 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:18:10.697 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:18:10.739 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:18:10.765 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:18:10.805 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:18:10.807 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:18:10.845 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:18:11.846 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:20:11.894 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:20:11.929 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:20:12.946 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:20:12.962 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:20:12.963 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:20:12.963 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:20:12.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:20:13.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:20:13.210 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:20:13.470 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:20:13.470 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:20:13.505 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:20:13.542 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:20:13.572 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:20:13.574 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:20:13.611 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:20:14.661 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:22:14.709 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:22:14.749 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:22:15.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:22:15.836 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:22:15.836 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:22:15.836 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:22:15.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:22:16.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:22:16.091 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:22:16.334 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:22:16.334 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:22:16.374 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:22:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:22:16.440 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:22:16.442 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:22:16.481 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:22:17.545 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:24:17.594 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:24:17.631 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:24:18.637 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:24:18.654 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:24:18.654 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:24:18.655 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:24:18.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:24:18.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:24:18.914 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:24:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:24:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:24:19.187 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:24:19.215 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:24:19.254 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:24:19.256 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:24:19.292 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:24:20.308 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:26:20.359 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:26:20.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:26:21.375 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:26:21.392 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:26:21.392 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:26:21.393 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:26:21.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:26:21.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:26:21.638 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:26:21.899 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:26:21.899 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:26:21.931 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:26:21.970 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:26:21.998 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:26:22.000 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:26:22.035 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:26:23.043 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:28:23.093 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:28:23.127 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:28:24.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:28:24.143 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:28:24.144 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:28:24.144 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:28:24.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:28:24.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:28:24.382 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:28:24.625 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:28:24.625 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:28:24.666 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:28:24.696 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:28:24.733 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:28:24.734 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:28:24.807 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:28:25.773 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:30:25.819 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:30:25.856 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:30:26.900 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:30:26.917 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:30:26.918 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:30:26.918 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:30:26.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:30:27.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:30:27.177 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:30:27.393 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:30:27.393 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:30:27.434 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:30:27.461 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:30:27.500 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:30:27.501 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:30:27.539 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:30:28.552 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:32:28.598 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:32:28.634 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:32:29.625 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:32:29.641 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:32:29.641 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:32:29.642 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:32:29.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:32:29.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:32:29.883 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:32:30.146 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:32:30.146 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:32:30.188 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:32:30.218 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:32:30.257 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:32:30.258 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:32:30.295 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:32:31.232 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:34:31.277 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:34:31.314 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:34:32.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:34:32.384 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:34:32.385 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:34:32.385 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:34:32.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:34:32.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:34:32.627 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:34:32.869 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:34:32.869 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:34:32.910 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:34:32.940 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:34:32.976 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:34:32.978 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:34:33.016 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:34:34.029 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:36:34.078 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:36:34.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:36:35.088 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:36:35.105 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:36:35.105 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:36:35.105 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:36:35.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:36:35.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:36:35.365 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:36:35.592 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:36:35.592 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:36:35.632 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:36:35.661 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:36:35.698 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:36:35.700 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:36:35.737 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:36:36.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:38:36.737 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:38:36.772 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:38:37.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:38:37.768 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:38:37.768 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:38:37.769 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:38:37.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:38:37.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:38:38.012 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:38:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:38:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:38:38.299 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:38:38.339 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:38:38.366 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:38:38.367 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:38:38.403 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:38:39.359 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:40:39.408 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:40:39.444 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:40:40.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:40:40.418 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:40:40.419 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:40:40.419 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:40:40.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:40:40.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:40:40.660 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:40:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:40:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:40:40.946 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:40:40.975 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:40:41.013 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:40:41.015 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:40:41.052 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:40:42.005 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:42:42.056 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:42:42.093 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:42:43.061 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:42:43.076 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:42:43.077 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:42:43.077 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:42:43.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:42:43.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:42:43.339 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:42:43.565 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:42:43.565 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:42:43.609 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:42:43.636 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:42:43.677 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:42:43.678 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:42:43.713 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:42:44.650 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:44:44.699 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:44:44.739 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:44:45.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:44:45.813 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:44:45.813 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:44:45.813 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:44:45.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:44:46.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:44:46.059 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:44:46.318 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:44:46.318 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:44:46.352 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:44:46.390 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:44:46.420 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:44:46.422 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:44:46.461 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:44:47.504 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:46:47.564 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:46:47.600 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:46:48.586 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:46:48.603 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:46:48.603 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:46:48.603 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:46:48.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:46:48.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:46:48.846 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:46:49.085 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:46:49.085 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:46:49.125 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:46:49.152 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:46:49.192 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:46:49.193 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:46:49.228 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:46:50.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:48:50.307 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:48:50.344 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:48:51.326 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:48:51.342 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:48:51.342 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:48:51.342 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:48:51.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:48:51.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:48:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:48:51.834 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:48:51.835 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:48:51.877 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:48:51.904 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:48:51.945 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:48:51.946 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:48:51.985 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:48:52.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:50:53.042 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:50:53.081 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:50:54.081 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:50:54.098 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:50:54.098 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:50:54.098 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:50:54.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:50:54.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:50:54.341 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:50:54.606 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:50:54.606 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:50:54.640 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:50:54.677 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:50:54.708 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:50:54.709 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:50:54.745 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:50:55.797 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:52:55.849 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:52:55.886 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:52:57.926 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:52:57.943 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:52:57.943 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:52:57.943 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:52:57.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:52:58.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:52:58.188 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:52:58.440 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:52:58.440 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:52:58.480 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:52:58.509 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:52:58.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:52:58.547 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:52:58.588 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:53:00.288 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:55:00.338 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:55:00.376 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:55:02.157 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:55:02.174 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:55:02.175 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:55:02.175 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:55:02.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:55:02.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:55:02.433 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:55:02.663 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:55:02.663 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:55:02.705 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:55:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:55:02.773 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:55:02.774 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:55:02.813 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:55:03.969 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:57:04.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:57:04.056 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:57:05.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:57:05.886 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:57:05.886 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:57:05.886 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:57:05.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:57:06.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:57:06.129 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:57:06.389 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:57:06.389 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:57:06.419 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:57:06.458 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:57:06.485 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:57:06.487 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:57:06.524 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:57:07.683 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:59:07.733 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T21:59:07.769 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:59:09.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T21:59:09.518 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T21:59:09.518 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T21:59:09.518 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:59:09.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:59:09.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T21:59:09.763 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T21:59:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T21:59:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T21:59:10.048 INFO:teuthology.misc:Compressing logs... 2021-03-20T21:59:10.077 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T21:59:10.116 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T21:59:10.117 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T21:59:10.157 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T21:59:12.028 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:01:12.096 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:01:12.136 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:01:13.989 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:01:14.007 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:01:14.008 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:01:14.008 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:01:14.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:01:14.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:01:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:01:14.504 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:01:14.505 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:01:14.547 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:01:14.574 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:01:14.614 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:01:14.616 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:01:14.657 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:01:15.972 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:03:16.031 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:03:16.069 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:03:17.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:03:17.865 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:03:17.865 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:03:17.865 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:03:17.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:03:18.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:03:18.106 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:03:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:03:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:03:18.392 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:03:18.433 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:03:18.458 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:03:18.460 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:03:18.499 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:03:19.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:05:19.551 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:05:19.587 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:05:20.636 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:05:20.653 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:05:20.654 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:05:20.654 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:05:20.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:05:20.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:05:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:05:21.143 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:05:21.143 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:05:21.183 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:05:21.213 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:05:21.251 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:05:21.252 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:05:21.288 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:05:22.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:07:22.317 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:07:22.352 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:07:23.374 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:07:23.390 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:07:23.390 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:07:23.391 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:07:23.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:07:23.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:07:23.649 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:07:23.872 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:07:23.872 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:07:23.913 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:07:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:07:23.980 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:07:23.981 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:07:24.017 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:07:24.970 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:09:25.023 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:09:25.058 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:09:26.008 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:09:26.028 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:09:26.028 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:09:26.028 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:09:26.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:09:26.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:09:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:09:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:09:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:09:26.573 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:09:26.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:09:26.640 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:09:26.641 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:09:26.677 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:09:27.654 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:11:27.703 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:11:27.739 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:11:28.733 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:11:28.750 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:11:28.751 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:11:28.751 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:11:28.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:11:28.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:11:28.994 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:11:29.243 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:11:29.243 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:11:29.284 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:11:29.313 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:11:29.351 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:11:29.353 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:11:29.393 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:11:30.328 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:13:30.394 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:13:30.432 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:13:31.448 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:13:31.465 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:13:31.465 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:13:31.465 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:13:31.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:13:31.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:13:31.723 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:13:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:13:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:13:31.998 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:13:32.025 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:13:32.066 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:13:32.068 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:13:32.105 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:13:33.120 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:15:33.168 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:15:33.206 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:15:34.241 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:15:34.259 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:15:34.259 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:15:34.259 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:15:34.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:15:34.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:15:34.504 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:15:34.763 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:15:34.763 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:15:34.795 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:15:34.831 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:15:34.861 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:15:34.863 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:15:34.899 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:15:35.901 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:17:35.948 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:17:35.984 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:17:36.911 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:17:36.927 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:17:36.927 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:17:36.927 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:17:36.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:17:37.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:17:37.166 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:17:37.399 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:17:37.399 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:17:37.442 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:17:37.467 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:17:37.511 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:17:37.512 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:17:37.546 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:17:38.559 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:19:38.610 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:19:38.645 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:19:39.629 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:19:39.648 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:19:39.649 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:19:39.649 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:19:39.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:19:39.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:19:39.908 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:19:40.136 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:19:40.136 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:19:40.176 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:19:40.205 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:19:40.244 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:19:40.245 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:19:40.278 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:19:41.268 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:21:41.314 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:21:41.349 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:21:42.326 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:21:42.345 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:21:42.345 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:21:42.346 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:21:42.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:21:42.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:21:42.589 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:21:42.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:21:42.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:21:42.883 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:21:42.922 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:21:42.951 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:21:42.952 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:21:42.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:21:43.989 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:23:44.047 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:23:44.084 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:23:45.125 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:23:45.144 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:23:45.144 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:23:45.144 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:23:45.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:23:45.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:23:45.396 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:23:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:23:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:23:45.686 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:23:45.715 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:23:45.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:23:45.755 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:23:45.792 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:23:46.853 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:25:46.909 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:25:46.946 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:25:47.917 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:25:47.933 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:25:47.933 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:25:47.933 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:25:47.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:25:48.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:25:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:25:48.421 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:25:48.422 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:25:48.464 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:25:48.490 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:25:48.533 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:25:48.535 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:25:48.570 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:25:49.524 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:27:49.573 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:27:49.607 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:27:50.565 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:27:50.584 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:27:50.585 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:27:50.585 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:27:50.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:27:50.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:27:50.827 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:27:51.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:27:51.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:27:51.126 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:27:51.162 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:27:51.195 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:27:51.195 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:27:51.231 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:27:52.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:29:52.257 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:29:52.294 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:29:53.293 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:29:53.310 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:29:53.310 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:29:53.310 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:29:53.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:29:53.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:29:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:29:53.794 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:29:53.794 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:29:53.833 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:29:53.863 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:29:53.899 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:29:53.900 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:29:53.939 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:29:54.920 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:31:54.976 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:31:55.013 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:31:56.010 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:31:56.027 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:31:56.027 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:31:56.027 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:31:56.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:31:56.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:31:56.289 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:31:56.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:31:56.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:31:56.560 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:31:56.590 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:31:56.627 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:31:56.629 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:31:56.663 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:31:57.626 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:33:57.676 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:33:57.714 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:33:58.737 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:33:58.758 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:33:58.758 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:33:58.758 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:33:58.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:33:58.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:33:59.003 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:33:59.261 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:33:59.261 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:33:59.290 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:33:59.332 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:33:59.356 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:33:59.358 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:33:59.396 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:34:00.441 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:36:00.491 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:36:00.525 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:36:01.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:36:01.570 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:36:01.571 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:36:01.571 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:36:01.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:36:01.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:36:01.812 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:36:02.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:36:02.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:36:02.099 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:36:02.129 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:36:02.166 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:36:02.167 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:36:02.201 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:36:03.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:38:03.260 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:38:03.344 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:38:04.422 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:38:04.438 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:38:04.438 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:38:04.438 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:38:04.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:38:04.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:38:04.696 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:38:04.933 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:38:04.934 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:38:04.973 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:38:05.002 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:38:05.041 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:38:05.042 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:38:05.078 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:38:06.131 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:40:06.180 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:40:06.216 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:40:07.268 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:40:07.287 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:40:07.287 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:40:07.287 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:40:07.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:40:07.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:40:07.532 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:40:07.796 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:40:07.796 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:40:07.829 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:40:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:40:07.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:40:07.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:40:07.934 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:40:09.006 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:42:09.059 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:42:09.094 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:42:10.055 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:42:10.071 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:42:10.071 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:42:10.071 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:42:10.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:42:10.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:42:10.318 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:42:10.561 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:42:10.562 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:42:10.603 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:42:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:42:10.671 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:42:10.672 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:42:10.710 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:42:11.729 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:44:11.778 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:44:11.816 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:44:12.867 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:44:12.883 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:44:12.883 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:44:12.884 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:44:12.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:44:13.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:44:13.142 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:44:13.369 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:44:13.370 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:44:13.410 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:44:13.438 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:44:13.477 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:44:13.478 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:44:13.512 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:44:14.484 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:46:14.533 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:46:14.568 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:46:16.383 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:46:16.400 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:46:16.400 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:46:16.401 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:46:16.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:46:16.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:46:16.655 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:46:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:46:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:46:16.949 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:46:16.989 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:46:17.015 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:46:17.018 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:46:17.053 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:46:18.030 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:48:18.079 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:48:18.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:48:19.945 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:48:19.962 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:48:19.963 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:48:19.963 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:48:19.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:48:20.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:48:20.233 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:48:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:48:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:48:20.547 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:48:20.581 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:48:20.617 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:48:20.619 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:48:20.658 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:48:22.228 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:50:22.277 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:50:22.314 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:50:23.328 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:50:23.346 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:50:23.346 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:50:23.346 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:50:23.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:50:23.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:50:23.605 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:50:23.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:50:23.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:50:23.875 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:50:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:50:23.942 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:50:23.944 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:50:23.981 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:50:24.989 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:52:25.036 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:52:25.070 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:52:26.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:52:26.062 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:52:26.062 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:52:26.062 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:52:26.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:52:26.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:52:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:52:26.565 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:52:26.566 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:52:26.595 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:52:26.636 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:52:26.662 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:52:26.663 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:52:26.697 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:52:27.654 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:54:27.705 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:54:27.739 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:54:28.652 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:54:28.668 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:54:28.668 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:54:28.668 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:54:28.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:54:28.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:54:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:54:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:54:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:54:29.196 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:54:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:54:29.263 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:54:29.265 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:54:29.299 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:54:30.232 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:56:30.284 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:56:30.328 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:56:31.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:56:31.448 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:56:31.448 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:56:31.448 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:56:31.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:56:31.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:56:31.710 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:56:31.937 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:56:31.937 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:56:31.979 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:56:32.006 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:56:32.045 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:56:32.047 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:56:32.086 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:56:33.173 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:58:33.225 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T22:58:33.261 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:58:34.217 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T22:58:34.233 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T22:58:34.233 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T22:58:34.233 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:58:34.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:58:34.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T22:58:34.476 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T22:58:34.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T22:58:34.731 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T22:58:34.760 INFO:teuthology.misc:Compressing logs... 2021-03-20T22:58:34.801 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T22:58:34.826 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T22:58:34.828 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T22:58:34.862 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T22:58:35.802 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:00:35.850 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:00:35.889 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:00:37.018 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:00:37.037 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:00:37.037 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:00:37.037 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:00:37.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:00:37.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:00:37.285 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:00:37.532 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:00:37.533 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:00:37.573 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:00:37.602 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:00:37.641 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:00:37.642 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:00:37.682 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:00:38.803 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:02:38.856 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:02:38.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:02:39.895 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:02:39.913 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:02:39.913 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:02:39.913 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:02:39.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:02:40.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:02:40.174 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:02:40.397 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:02:40.397 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:02:40.437 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:02:40.465 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:02:40.504 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:02:40.506 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:02:40.540 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:02:41.538 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:04:41.587 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:04:41.621 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:04:42.574 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:04:42.602 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:04:42.603 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:04:42.603 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:04:42.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:04:42.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:04:42.847 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:04:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:04:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:04:43.144 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:04:43.181 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:04:43.214 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:04:43.214 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:04:43.247 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:04:44.177 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:06:44.225 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:06:44.260 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:06:45.227 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:06:45.243 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:06:45.243 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:06:45.243 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:06:45.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:06:45.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:06:45.486 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:06:45.726 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:06:45.726 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:06:45.766 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:06:45.796 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:06:45.832 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:06:45.834 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:06:45.867 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:06:46.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:08:46.839 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:08:46.875 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:08:47.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:08:47.868 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:08:47.868 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:08:47.868 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:08:47.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:08:48.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:08:48.126 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:08:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:08:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:08:48.395 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:08:48.422 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:08:48.463 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:08:48.464 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:08:48.498 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:08:49.443 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:10:49.489 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:10:49.523 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:10:50.471 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:10:50.487 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:10:50.487 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:10:50.487 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:10:50.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:10:50.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:10:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:10:50.989 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:10:50.989 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:10:51.018 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:10:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:10:51.086 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:10:51.087 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:10:51.159 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:10:52.086 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:12:52.132 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:12:52.168 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:12:53.135 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:12:53.153 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:12:53.153 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:12:53.153 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:12:53.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:12:53.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:12:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:12:53.645 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:12:53.645 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:12:53.685 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:12:53.716 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:12:53.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:12:53.754 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:12:53.791 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:12:54.761 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:14:54.808 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:14:54.848 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:14:55.884 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:14:55.901 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:14:55.901 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:14:55.901 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:14:55.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:14:56.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:14:56.164 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:14:56.395 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:14:56.396 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:14:56.439 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:14:56.467 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:14:56.507 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:14:56.508 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:14:56.545 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:14:57.644 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:16:57.698 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:16:57.733 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:16:58.683 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:16:58.699 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:16:58.699 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:16:58.700 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:16:58.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:16:58.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:16:58.944 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:16:59.200 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:16:59.201 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:16:59.230 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:16:59.270 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:16:59.297 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:16:59.299 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:16:59.334 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:17:00.254 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:19:00.302 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:19:00.337 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:19:01.267 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:19:01.283 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:19:01.283 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:19:01.284 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:19:01.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:19:01.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:19:01.524 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:19:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:19:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:19:01.806 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:19:01.834 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:19:01.874 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:19:01.876 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:19:01.911 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:19:02.896 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:21:02.944 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:21:02.978 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:21:03.917 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:21:03.934 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:21:03.934 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:21:03.934 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:21:03.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:21:04.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:21:04.192 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:21:04.418 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:21:04.418 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:21:04.462 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:21:04.489 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:21:04.530 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:21:04.531 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:21:04.564 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:21:05.477 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:23:05.522 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:23:05.557 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:23:06.517 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:23:06.532 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:23:06.533 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:23:06.533 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:23:06.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:23:06.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:23:06.779 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:23:07.035 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:23:07.036 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:23:07.064 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:23:07.104 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:23:07.131 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:23:07.132 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:23:07.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:23:08.099 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:25:08.147 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:25:08.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:25:09.123 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:25:09.139 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:25:09.140 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:25:09.140 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:25:09.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:25:09.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:25:09.390 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:25:09.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:25:09.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:25:09.675 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:25:09.705 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:25:09.742 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:25:09.744 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:25:09.777 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:25:10.698 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:27:10.745 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:27:10.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:27:11.749 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:27:11.765 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:27:11.765 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:27:11.765 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:27:11.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:27:11.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:27:12.026 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:27:12.253 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:27:12.253 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:27:12.295 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:27:12.323 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:27:12.362 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:27:12.363 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:27:12.399 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:27:13.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:29:13.400 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:29:13.435 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:29:14.434 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:29:14.450 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:29:14.450 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:29:14.450 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:29:14.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:29:14.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:29:14.692 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:29:14.954 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:29:14.954 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:29:14.983 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:29:15.022 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:29:15.050 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:29:15.052 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:29:15.085 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:29:16.036 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:31:16.096 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:31:16.130 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:31:17.136 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:31:17.153 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:31:17.153 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:31:17.154 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:31:17.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:31:17.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:31:17.405 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:31:17.650 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:31:17.650 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:31:17.690 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:31:17.722 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:31:17.756 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:31:17.758 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:31:17.794 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:31:18.806 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:33:18.852 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:33:18.888 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:33:19.836 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:33:19.853 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:33:19.853 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:33:19.853 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:33:19.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:33:20.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:33:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:33:20.335 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:33:20.335 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:33:20.377 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:33:20.403 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:33:20.444 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:33:20.446 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:33:20.482 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:33:21.381 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:35:21.427 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:35:21.464 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:35:22.487 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:35:22.504 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:35:22.504 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:35:22.504 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:35:22.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:35:22.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:35:22.744 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:35:23.001 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:35:23.001 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:35:23.031 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:35:23.070 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:35:23.097 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:35:23.099 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:35:23.133 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:35:24.097 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:37:24.147 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:37:24.180 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:37:25.230 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:37:25.246 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:37:25.247 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:37:25.247 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:37:25.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:37:25.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:37:25.491 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:37:25.732 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:37:25.733 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:37:25.774 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:37:25.802 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:37:25.841 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:37:25.842 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:37:25.879 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:37:26.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:39:26.864 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:39:26.901 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:39:27.872 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:39:27.889 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:39:27.889 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:39:27.890 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:39:27.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:39:28.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:39:28.147 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:39:28.383 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:39:28.383 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:39:28.425 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:39:28.452 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:39:28.494 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:39:28.496 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:39:28.529 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:39:29.470 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:41:29.516 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:41:29.550 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:41:30.494 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:41:30.511 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:41:30.511 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:41:30.511 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:41:30.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:41:30.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:41:30.754 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:41:31.011 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:41:31.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:41:31.041 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:41:31.080 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:41:31.109 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:41:31.111 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:41:31.144 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:41:32.085 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:43:32.133 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:43:32.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:43:33.146 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:43:33.164 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:43:33.164 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:43:33.164 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:43:33.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:43:33.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:43:33.406 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:43:33.653 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:43:33.653 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:43:33.693 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:43:33.724 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:43:33.760 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:43:33.761 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:43:33.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:43:34.711 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:45:34.768 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:45:34.851 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:45:35.872 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:45:35.888 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:45:35.888 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:45:35.888 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:45:35.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:45:36.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:45:36.147 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:45:36.377 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:45:36.377 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:45:36.419 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:45:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:45:36.486 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:45:36.488 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:45:36.523 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:45:37.511 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:47:37.559 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:47:37.591 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:47:38.657 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:47:38.674 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:47:38.674 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:47:38.675 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:47:38.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:47:38.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:47:38.918 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:47:39.175 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:47:39.175 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:47:39.205 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:47:39.244 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:47:39.271 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:47:39.273 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:47:39.311 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:47:40.299 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:49:40.347 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:49:40.380 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:49:41.302 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:49:41.320 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:49:41.320 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:49:41.320 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:49:41.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:49:41.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:49:41.563 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:49:41.811 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:49:41.811 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:49:41.853 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:49:41.880 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:49:41.920 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:49:41.921 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:49:41.957 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:49:42.867 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:51:42.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:51:42.946 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:51:43.881 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:51:43.896 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:51:43.896 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:51:43.896 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:51:43.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:51:44.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:51:44.157 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:51:44.383 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:51:44.383 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:51:44.425 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:51:44.451 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:51:44.492 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:51:44.493 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:51:44.526 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:51:45.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:53:45.514 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:53:45.551 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:53:47.356 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:53:47.372 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:53:47.372 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:53:47.372 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:53:47.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:53:47.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:53:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:53:47.869 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:53:47.869 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:53:47.900 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:53:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:53:47.967 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:53:47.968 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:53:48.003 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:53:49.100 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:55:49.148 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:55:49.181 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:55:50.270 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:55:50.287 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:55:50.287 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:55:50.287 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:55:50.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:55:50.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:55:50.531 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:55:50.776 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:55:50.776 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:55:50.817 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:55:50.844 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:55:50.884 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:55:50.886 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:55:50.923 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:55:51.964 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:57:52.011 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:57:52.045 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:57:53.809 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:57:53.825 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:57:53.826 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:57:53.826 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:57:53.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:57:54.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:57:54.084 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:57:54.310 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:57:54.310 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:57:54.352 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:57:54.380 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:57:54.419 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:57:54.420 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:57:54.457 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:57:55.790 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:59:55.854 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-20T23:59:55.889 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:59:57.764 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-20T23:59:57.780 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-20T23:59:57.781 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-20T23:59:57.781 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:59:57.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:59:57.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:59:58.024 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-20T23:59:58.289 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-20T23:59:58.289 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-20T23:59:58.320 INFO:teuthology.misc:Compressing logs... 2021-03-20T23:59:58.358 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-20T23:59:58.388 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-20T23:59:58.390 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-20T23:59:58.427 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-20T23:59:59.830 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:01:59.879 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:01:59.928 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:02:01.984 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:02:02.002 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:02:02.002 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:02:02.002 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:02:02.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:02:02.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:02:02.249 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:02:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:02:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:02:02.530 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:02:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:02:02.597 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:02:02.599 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:02:02.679 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:02:04.272 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:04:04.324 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:04:04.364 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:04:06.360 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:04:06.379 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:04:06.379 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:04:06.379 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:04:06.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:04:06.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:04:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:04:06.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:04:06.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:04:06.926 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:04:06.954 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:04:06.993 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:04:06.995 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:04:07.036 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:04:08.401 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:06:08.452 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:06:08.495 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:06:10.333 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:06:10.349 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:06:10.349 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:06:10.349 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:06:10.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:06:10.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:06:10.595 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:06:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:06:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:06:10.884 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:06:10.924 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:06:10.951 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:06:10.952 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:06:10.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:06:11.969 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:08:12.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:08:12.054 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:08:13.158 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:08:13.174 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:08:13.174 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:08:13.174 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:08:13.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:08:13.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:08:13.418 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:08:13.665 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:08:13.665 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:08:13.704 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:08:13.738 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:08:13.771 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:08:13.772 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:08:13.809 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:08:14.728 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:10:14.778 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:10:14.812 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:10:15.844 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:10:15.860 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:10:15.860 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:10:15.860 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:10:15.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:10:16.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:10:16.114 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:10:16.342 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:10:16.342 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:10:16.385 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:10:16.412 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:10:16.454 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:10:16.455 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:10:16.491 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:10:17.467 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:12:17.519 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:12:17.554 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:12:18.674 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:12:18.690 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:12:18.690 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:12:18.690 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:12:18.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:12:18.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:12:18.932 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:12:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:12:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:12:19.219 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:12:19.260 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:12:19.286 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:12:19.287 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:12:19.328 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:12:20.255 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:14:20.307 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:14:20.343 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:14:21.363 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:14:21.379 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:14:21.379 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:14:21.379 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:14:21.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:14:21.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:14:21.624 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:14:21.866 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:14:21.866 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:14:21.907 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:14:21.935 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:14:21.973 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:14:21.975 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:14:22.009 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:14:22.956 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:16:23.003 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:16:23.041 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:16:24.079 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:16:24.096 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:16:24.096 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:16:24.096 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:16:24.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:16:24.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:16:24.358 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:16:24.583 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:16:24.583 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:16:24.622 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:16:24.653 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:16:24.689 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:16:24.690 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:16:24.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:16:25.731 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:18:25.780 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:18:25.818 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:18:27.725 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:18:27.742 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:18:27.743 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:18:27.743 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:18:27.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:18:27.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:18:27.987 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:18:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:18:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:18:28.267 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:18:28.306 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:18:28.333 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:18:28.334 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:18:28.419 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:18:29.477 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:20:29.528 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:20:29.567 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:20:30.523 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:20:30.538 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:20:30.538 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:20:30.538 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:20:30.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:20:30.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:20:30.781 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:20:31.025 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:20:31.025 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:20:31.065 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:20:31.096 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:20:31.131 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:20:31.133 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:20:31.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:20:32.078 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:22:32.138 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:22:32.175 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:22:33.124 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:22:33.143 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:22:33.143 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:22:33.143 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:22:33.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:22:33.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:22:33.399 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:22:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:22:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:22:33.672 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:22:33.699 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:22:33.740 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:22:33.741 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:22:33.778 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:22:34.773 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:24:34.821 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:24:34.855 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:24:35.839 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:24:35.855 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:24:35.855 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:24:35.856 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:24:35.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:24:36.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:24:36.094 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:24:36.355 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:24:36.355 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:24:36.385 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:24:36.424 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:24:36.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:24:36.453 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:24:36.487 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:24:37.432 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:26:37.479 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:26:37.516 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:26:38.521 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:26:38.539 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:26:38.539 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:26:38.539 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:26:38.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:26:38.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:26:38.787 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:26:39.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:26:39.031 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:26:39.070 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:26:39.102 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:26:39.136 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:26:39.138 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:26:39.172 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:26:40.121 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:28:40.170 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:28:40.207 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:28:41.209 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:28:41.227 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:28:41.227 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:28:41.227 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:28:41.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:28:41.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:28:41.488 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:28:41.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:28:41.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:28:41.753 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:28:41.780 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:28:41.819 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:28:41.820 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:28:41.855 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:28:42.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:30:42.824 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:30:42.858 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:30:44.061 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:30:44.077 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:30:44.077 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:30:44.078 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:30:44.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:30:44.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:30:44.321 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:30:44.580 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:30:44.580 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:30:44.610 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:30:44.649 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:30:44.676 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:30:44.677 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:30:44.712 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:30:45.670 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:32:45.720 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:32:45.756 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:32:46.819 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:32:46.836 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:32:46.836 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:32:46.837 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:32:46.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:32:47.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:32:47.081 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:32:47.323 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:32:47.323 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:32:47.364 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:32:47.392 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:32:47.431 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:32:47.433 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:32:47.467 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:32:48.461 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:34:48.509 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:34:48.544 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:34:49.595 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:34:49.630 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:34:49.630 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:34:49.630 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:34:49.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:34:49.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:34:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:34:50.127 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:34:50.127 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:34:50.169 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:34:50.195 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:34:50.236 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:34:50.238 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:34:50.274 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:34:51.218 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:36:51.269 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:36:51.301 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:36:52.552 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:36:52.569 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:36:52.569 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:36:52.569 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:36:52.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:36:52.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:36:52.815 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:36:53.072 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:36:53.072 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:36:53.102 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:36:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:36:53.168 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:36:53.170 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:36:53.209 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:36:54.170 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:38:54.219 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:38:54.256 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:38:55.291 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:38:55.307 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:38:55.307 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:38:55.307 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:38:55.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:38:55.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:38:55.552 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:38:55.799 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:38:55.800 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:38:55.840 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:38:55.868 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:38:55.908 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:38:55.909 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:38:55.942 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:38:56.847 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:40:56.897 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:40:56.932 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:40:58.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:40:58.028 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:40:58.028 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:40:58.029 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:40:58.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:40:58.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:40:58.290 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:40:58.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:40:58.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:40:58.562 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:40:58.589 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:40:58.630 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:40:58.631 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:40:58.666 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:40:59.638 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:42:59.687 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:42:59.722 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:43:00.773 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:43:00.788 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:43:00.788 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:43:00.788 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:43:00.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:43:01.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:43:01.032 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:43:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:43:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:43:01.322 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:43:01.361 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:43:01.391 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:43:01.392 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:43:01.428 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:43:02.368 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:45:02.419 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:45:02.455 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:45:03.499 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:45:03.515 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:45:03.515 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:45:03.516 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:45:03.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:45:03.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:45:03.759 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:45:04.002 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:45:04.002 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:45:04.042 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:45:04.072 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:45:04.108 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:45:04.110 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:45:04.149 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:45:05.097 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:47:05.147 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:47:05.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:47:06.232 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:47:06.251 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:47:06.251 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:47:06.251 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:47:06.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:47:06.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:47:06.511 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:47:06.743 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:47:06.743 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:47:06.785 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:47:06.812 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:47:06.852 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:47:06.854 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:47:06.889 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:47:07.830 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:49:07.879 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:49:07.913 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:49:08.878 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:49:08.894 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:49:08.894 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:49:08.894 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:49:08.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:49:09.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:49:09.133 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:49:09.393 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:49:09.393 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:49:09.423 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:49:09.461 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:49:09.490 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:49:09.491 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:49:09.525 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:49:10.465 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:51:10.516 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:51:10.549 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:51:11.511 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:51:11.526 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:51:11.527 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:51:11.527 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:51:11.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:51:11.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:51:11.771 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:51:12.016 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:51:12.016 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:51:12.057 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:51:12.084 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:51:12.124 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:51:12.125 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:51:12.159 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:51:13.104 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:53:13.151 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:53:13.187 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:53:14.134 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:53:14.152 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:53:14.152 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:53:14.152 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:53:14.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:53:14.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:53:14.411 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:53:14.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:53:14.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:53:14.677 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:53:14.703 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:53:14.744 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:53:14.745 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:53:14.817 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:53:15.749 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:55:15.797 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:55:15.830 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:55:16.773 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:55:16.788 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:55:16.789 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:55:16.789 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:55:16.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:55:17.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:55:17.032 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:55:17.289 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:55:17.289 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:55:17.318 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:55:17.359 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:55:17.384 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:55:17.386 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:55:17.420 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:55:18.352 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:57:18.401 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:57:18.437 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:57:19.432 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:57:19.447 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:57:19.448 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:57:19.448 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:57:19.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:57:19.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:57:19.688 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:57:19.936 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:57:19.936 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:57:19.975 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:57:20.006 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:57:20.041 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:57:20.042 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:57:20.076 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:57:20.984 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:59:21.032 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T00:59:21.065 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:59:22.008 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T00:59:22.024 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T00:59:22.024 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T00:59:22.024 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:59:22.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:59:22.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T00:59:22.281 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T00:59:22.518 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T00:59:22.518 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T00:59:22.561 INFO:teuthology.misc:Compressing logs... 2021-03-21T00:59:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T00:59:22.629 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T00:59:22.631 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T00:59:22.665 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T00:59:23.645 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:01:23.698 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:01:23.737 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:01:24.806 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:01:24.825 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:01:24.825 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:01:24.825 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:01:24.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:01:25.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:01:25.072 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:01:25.338 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:01:25.339 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:01:25.374 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:01:25.407 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:01:25.442 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:01:25.444 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:01:25.482 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:01:26.573 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:03:26.627 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:03:26.663 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:03:27.691 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:03:27.707 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:03:27.708 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:03:27.708 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:03:27.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:03:27.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:03:27.953 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:03:28.196 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:03:28.196 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:03:28.234 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:03:28.267 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:03:28.300 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:03:28.302 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:03:28.337 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:03:29.321 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:05:29.374 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:05:29.412 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:05:30.542 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:05:30.558 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:05:30.558 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:05:30.558 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:05:30.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:05:30.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:05:30.820 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:05:31.050 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:05:31.050 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:05:31.092 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:05:31.119 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:05:31.159 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:05:31.161 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:05:31.197 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:05:32.207 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:07:32.260 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:07:32.298 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:07:34.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:07:34.257 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:07:34.257 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:07:34.257 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:07:34.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:07:34.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:07:34.501 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:07:34.757 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:07:34.757 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:07:34.787 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:07:34.826 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:07:34.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:07:34.855 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:07:34.894 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:07:35.972 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:09:36.023 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:09:36.064 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:09:37.948 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:09:37.966 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:09:37.967 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:09:37.967 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:09:37.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:09:38.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:09:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:09:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:09:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:09:38.497 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:09:38.523 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:09:38.564 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:09:38.566 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:09:38.604 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:09:39.614 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:11:39.671 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:11:39.709 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:11:40.733 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:11:40.751 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:11:40.751 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:11:40.751 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:11:40.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:11:40.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:11:41.007 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:11:41.238 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:11:41.238 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:11:41.279 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:11:41.305 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:11:41.345 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:11:41.347 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:11:41.383 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:11:42.402 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:13:42.456 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:13:42.497 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:13:43.612 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:13:43.628 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:13:43.628 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:13:43.628 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:13:43.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:13:43.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:13:43.876 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:13:44.139 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:13:44.139 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:13:44.172 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:13:44.211 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:13:44.241 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:13:44.241 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:13:44.280 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:13:45.396 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:15:45.449 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:15:45.488 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:15:46.610 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:15:46.627 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:15:46.627 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:15:46.627 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:15:46.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:15:46.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:15:46.873 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:15:47.115 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:15:47.115 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:15:47.156 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:15:47.183 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:15:47.224 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:15:47.226 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:15:47.261 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:15:48.222 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:17:48.271 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:17:48.309 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:17:49.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:17:49.352 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:17:49.353 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:17:49.353 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:17:49.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:17:49.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:17:49.614 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:17:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:17:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:17:49.885 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:17:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:17:49.952 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:17:49.954 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:17:49.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:17:50.961 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:19:51.010 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:19:51.048 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:19:52.105 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:19:52.120 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:19:52.120 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:19:52.120 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:19:52.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:19:52.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:19:52.359 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:19:52.616 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:19:52.616 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:19:52.645 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:19:52.685 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:19:52.711 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:19:52.712 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:19:52.749 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:19:53.780 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:21:53.831 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:21:53.866 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:21:54.887 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:21:54.903 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:21:54.903 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:21:54.903 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:21:54.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:21:55.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:21:55.147 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:21:55.389 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:21:55.389 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:21:55.430 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:21:55.458 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:21:55.497 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:21:55.498 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:21:55.534 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:21:56.526 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:23:56.577 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:23:56.612 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:23:57.649 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:23:57.665 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:23:57.665 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:23:57.665 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:23:57.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:23:57.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:23:57.924 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:23:58.148 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:23:58.148 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:23:58.190 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:23:58.217 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:23:58.257 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:23:58.258 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:23:58.291 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:23:59.299 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:25:59.348 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:25:59.385 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:26:00.395 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:26:00.413 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:26:00.413 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:26:00.413 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:26:00.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:26:00.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:26:00.656 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:26:00.915 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:26:00.915 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:26:00.946 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:26:00.985 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:26:01.012 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:26:01.014 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:26:01.051 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:26:02.102 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:28:02.153 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:28:02.233 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:28:03.250 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:28:03.266 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:28:03.266 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:28:03.266 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:28:03.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:28:03.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:28:03.508 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:28:03.750 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:28:03.751 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:28:03.790 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:28:03.821 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:28:03.857 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:28:03.858 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:28:03.892 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:28:04.933 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:30:04.986 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:30:05.024 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:30:06.072 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:30:06.088 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:30:06.089 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:30:06.089 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:30:06.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:30:06.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:30:06.346 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:30:06.573 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:30:06.574 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:30:06.617 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:30:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:30:06.683 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:30:06.685 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:30:06.721 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:30:07.759 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:32:07.815 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:32:07.848 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:32:08.821 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:32:08.837 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:32:08.837 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:32:08.837 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:32:08.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:32:09.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:32:09.080 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:32:09.335 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:32:09.336 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:32:09.365 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:32:09.404 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:32:09.431 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:32:09.433 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:32:09.468 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:32:10.439 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:34:10.501 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:34:10.535 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:34:11.487 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:34:11.502 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:34:11.502 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:34:11.502 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:34:11.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:34:11.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:34:11.742 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:34:11.984 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:34:11.984 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:34:12.027 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:34:12.053 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:34:12.094 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:34:12.095 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:34:12.129 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:34:13.063 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:36:13.111 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:36:13.145 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:36:14.077 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:36:14.093 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:36:14.093 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:36:14.093 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:36:14.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:36:14.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:36:14.353 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:36:14.582 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:36:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:36:14.624 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:36:14.652 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:36:14.691 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:36:14.692 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:36:14.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:36:15.629 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:38:15.677 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:38:15.711 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:38:16.667 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:38:16.682 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:38:16.682 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:38:16.682 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:38:16.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:38:16.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:38:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:38:17.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:38:17.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:38:17.213 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:38:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:38:17.279 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:38:17.281 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:38:17.315 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:38:18.220 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:40:18.266 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:40:18.304 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:40:19.287 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:40:19.303 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:40:19.303 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:40:19.303 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:40:19.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:40:19.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:40:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:40:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:40:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:40:19.829 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:40:19.856 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:40:19.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:40:19.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:40:19.931 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:40:20.885 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:42:20.935 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:42:20.972 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:42:22.000 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:42:22.018 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:42:22.018 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:42:22.018 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:42:22.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:42:22.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:42:22.279 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:42:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:42:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:42:22.544 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:42:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:42:22.610 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:42:22.612 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:42:22.649 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:42:23.673 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:44:23.720 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:44:23.755 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:44:25.781 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:44:25.796 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:44:25.797 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:44:25.797 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:44:25.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:44:26.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:44:26.050 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:44:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:44:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:44:26.341 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:44:26.376 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:44:26.409 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:44:26.410 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:44:26.445 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:44:27.364 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:46:27.412 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:46:27.447 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:46:29.330 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:46:29.346 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:46:29.346 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:46:29.346 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:46:29.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:46:29.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:46:29.590 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:46:29.835 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:46:29.835 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:46:29.877 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:46:29.903 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:46:29.944 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:46:29.946 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:46:29.981 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:46:30.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:48:31.048 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:48:31.081 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:48:32.100 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:48:32.118 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:48:32.118 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:48:32.118 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:48:32.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:48:32.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:48:32.376 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:48:32.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:48:32.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:48:32.643 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:48:32.671 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:48:32.710 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:48:32.712 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:48:32.748 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:48:33.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:50:33.766 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:50:33.804 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:50:35.539 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:50:35.555 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:50:35.556 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:50:35.556 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:50:35.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:50:35.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:50:35.800 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:50:36.057 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:50:36.058 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:50:36.088 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:50:36.128 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:50:36.155 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:50:36.156 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:50:36.191 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:50:37.157 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:52:37.204 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:52:37.239 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:52:38.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:52:38.905 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:52:38.905 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:52:38.905 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:52:38.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:52:39.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:52:39.148 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:52:39.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:52:39.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:52:39.432 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:52:39.460 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:52:39.499 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:52:39.500 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:52:39.534 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:52:40.485 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:54:40.536 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:54:40.569 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:54:41.535 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:54:41.553 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:54:41.553 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:54:41.553 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:54:41.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:54:41.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:54:41.812 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:54:42.039 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:54:42.039 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:54:42.079 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:54:42.109 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:54:42.146 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:54:42.147 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:54:42.179 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:54:43.073 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:56:43.120 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:56:43.155 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:56:44.939 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:56:44.954 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:56:44.955 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:56:44.955 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:56:44.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:56:45.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:56:45.198 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:56:45.464 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:56:45.464 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:56:45.499 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:56:45.536 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:56:45.567 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:56:45.568 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:56:45.609 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:56:46.578 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:58:46.630 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T01:58:46.665 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:58:48.538 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T01:58:48.554 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T01:58:48.554 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T01:58:48.554 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:58:48.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:58:48.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T01:58:48.799 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T01:58:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T01:58:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T01:58:49.087 INFO:teuthology.misc:Compressing logs... 2021-03-21T01:58:49.117 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T01:58:49.155 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T01:58:49.157 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T01:58:49.195 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T01:58:50.165 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:00:50.211 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:00:50.252 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:00:52.377 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:00:52.394 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:00:52.395 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:00:52.395 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:00:52.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:00:52.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:00:52.656 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:00:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:00:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:00:52.927 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:00:52.953 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:00:52.993 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:00:52.995 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:00:53.079 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:00:54.175 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:02:54.229 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:02:54.270 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:02:56.324 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:02:56.345 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:02:56.345 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:02:56.345 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:02:56.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:02:56.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:02:56.604 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:02:56.864 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:02:56.864 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:02:56.895 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:02:56.934 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:02:56.963 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:02:56.964 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:02:57.010 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:02:58.431 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:04:58.483 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:04:58.517 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:05:00.497 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:05:00.513 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:05:00.513 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:05:00.513 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:05:00.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:05:00.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:05:00.760 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:05:01.002 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:05:01.003 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:05:01.044 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:05:01.072 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:05:01.111 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:05:01.113 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:05:01.148 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:05:02.238 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:07:02.288 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:07:02.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:07:03.438 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:07:03.454 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:07:03.454 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:07:03.454 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:07:03.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:07:03.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:07:03.716 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:07:03.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:07:03.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:07:03.993 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:07:04.021 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:07:04.061 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:07:04.062 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:07:04.098 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:07:05.039 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:09:05.088 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:09:05.122 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:09:06.139 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:09:06.156 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:09:06.156 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:09:06.156 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:09:06.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:09:06.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:09:06.400 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:09:06.657 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:09:06.657 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:09:06.687 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:09:06.728 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:09:06.755 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:09:06.756 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:09:06.794 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:09:07.741 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:11:07.788 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:11:07.826 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:11:08.856 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:11:08.872 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:11:08.873 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:11:08.873 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:11:08.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:11:09.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:11:09.120 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:11:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:11:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:11:09.412 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:11:09.444 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:11:09.480 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:11:09.481 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:11:09.516 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:11:10.465 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:13:10.518 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:13:10.555 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:13:11.587 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:13:11.602 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:13:11.602 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:13:11.602 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:13:11.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:13:11.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:13:11.859 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:13:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:13:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:13:12.124 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:13:12.150 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:13:12.192 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:13:12.194 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:13:12.229 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:13:13.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:15:13.216 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:15:13.249 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:15:14.255 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:15:14.270 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:15:14.270 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:15:14.270 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:15:14.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:15:14.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:15:14.516 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:15:14.779 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:15:14.780 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:15:14.813 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:15:14.849 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:15:14.880 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:15:14.883 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:15:14.920 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:15:15.876 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:17:15.929 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:17:15.965 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:17:17.046 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:17:17.066 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:17:17.066 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:17:17.067 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:17:17.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:17:17.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:17:17.306 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:17:17.549 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:17:17.550 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:17:17.594 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:17:17.620 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:17:17.661 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:17:17.663 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:17:17.704 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:17:18.695 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:19:18.746 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:19:18.783 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:19:19.755 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:19:19.771 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:19:19.771 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:19:19.771 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:19:19.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:19:19.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:19:20.029 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:19:20.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:19:20.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:19:20.279 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:19:20.309 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:19:20.345 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:19:20.346 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:19:20.380 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:19:21.294 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:21:21.341 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:21:21.378 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:21:22.397 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:21:22.413 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:21:22.413 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:21:22.413 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:21:22.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:21:22.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:21:22.657 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:21:22.912 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:21:22.912 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:21:22.941 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:21:22.982 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:21:23.007 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:21:23.009 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:21:23.041 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:21:24.038 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:23:24.087 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:23:24.121 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:23:25.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:23:25.222 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:23:25.222 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:23:25.222 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:23:25.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:23:25.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:23:25.465 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:23:25.707 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:23:25.708 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:23:25.750 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:23:25.776 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:23:25.816 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:23:25.818 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:23:25.852 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:23:26.779 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:25:26.828 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:25:26.895 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:25:27.916 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:25:27.932 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:25:27.932 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:25:27.932 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:25:27.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:25:28.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:25:28.193 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:25:28.420 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:25:28.421 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:25:28.463 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:25:28.490 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:25:28.530 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:25:28.532 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:25:28.565 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:25:29.509 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:27:29.565 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:27:29.600 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:27:30.637 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:27:30.652 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:27:30.653 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:27:30.653 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:27:30.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:27:30.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:27:30.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:27:31.154 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:27:31.155 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:27:31.184 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:27:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:27:31.251 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:27:31.253 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:27:31.288 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:27:32.226 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:29:32.276 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:29:32.314 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:29:33.399 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:29:33.416 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:29:33.416 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:29:33.416 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:29:33.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:29:33.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:29:33.659 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:29:33.904 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:29:33.904 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:29:33.945 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:29:33.974 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:29:34.012 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:29:34.014 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:29:34.051 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:29:35.065 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:31:35.111 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:31:35.146 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:31:36.122 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:31:36.138 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:31:36.138 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:31:36.138 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:31:36.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:31:36.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:31:36.394 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:31:36.622 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:31:36.622 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:31:36.662 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:31:36.691 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:31:36.729 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:31:36.731 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:31:36.764 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:31:37.631 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:33:37.683 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:33:37.720 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:33:38.748 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:33:38.763 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:33:38.763 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:33:38.764 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:33:38.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:33:38.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:33:39.008 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:33:39.267 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:33:39.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:33:39.298 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:33:39.339 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:33:39.367 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:33:39.368 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:33:39.402 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:33:40.332 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:35:40.383 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:35:40.417 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:35:42.228 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:35:42.290 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:35:42.290 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:35:42.290 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:35:42.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:35:42.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:35:42.527 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:35:42.770 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:35:42.771 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:35:42.812 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:35:42.839 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:35:42.879 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:35:42.881 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:35:42.915 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:35:43.862 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:37:43.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:37:43.949 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:37:45.670 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:37:45.687 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:37:45.687 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:37:45.687 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:37:45.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:37:45.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:37:45.954 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:37:46.186 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:37:46.186 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:37:46.227 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:37:46.255 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:37:46.294 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:37:46.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:37:46.336 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:37:47.415 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:39:47.464 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:39:47.499 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:39:49.174 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:39:49.190 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:39:49.191 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:39:49.191 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:39:49.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:39:49.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:39:49.440 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:39:49.696 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:39:49.696 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:39:49.726 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:39:49.793 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:39:49.795 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:39:49.832 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:39:50.772 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:41:50.820 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:41:50.856 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:41:52.433 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:41:52.452 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:41:52.452 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:41:52.452 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:41:52.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:41:52.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:41:52.700 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:41:52.944 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:41:52.944 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:41:52.986 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:41:53.015 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:41:53.053 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:41:53.055 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:41:53.092 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:41:54.130 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:43:54.179 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:43:54.215 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:43:55.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:43:55.247 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:43:55.248 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:43:55.248 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:43:55.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:43:55.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:43:55.511 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:43:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:43:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:43:55.785 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:43:55.812 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:43:55.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:43:55.855 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:43:55.892 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:43:56.903 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:45:56.951 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:45:56.988 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:45:58.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:45:58.215 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:45:58.215 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:45:58.215 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:45:58.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:45:58.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:45:58.461 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:45:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:45:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:45:58.757 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:45:58.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:45:58.824 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:45:58.826 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:45:58.859 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:45:59.839 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:47:59.886 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:47:59.921 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:48:00.897 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:48:00.915 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:48:00.915 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:48:00.915 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:48:00.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:48:01.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:48:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:48:01.399 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:48:01.400 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:48:01.439 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:48:01.469 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:48:01.506 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:48:01.508 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:48:01.544 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:48:02.508 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:50:02.560 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:50:02.597 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:50:03.658 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:50:03.692 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:50:03.692 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:50:03.693 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:50:03.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:50:03.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:50:03.957 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:50:04.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:50:04.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:50:04.225 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:50:04.251 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:50:04.291 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:50:04.293 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:50:04.329 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:50:05.337 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:52:05.384 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:52:05.457 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:52:07.514 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:52:07.531 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:52:07.531 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:52:07.531 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:52:07.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:52:07.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:52:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:52:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:52:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:52:08.062 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:52:08.102 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:52:08.128 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:52:08.129 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:52:08.166 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:52:09.196 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:54:09.245 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:54:09.279 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:54:11.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:54:11.287 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:54:11.287 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:54:11.288 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:54:11.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:54:11.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:54:11.531 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:54:11.776 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:54:11.776 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:54:11.817 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:54:11.844 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:54:11.884 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:54:11.885 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:54:11.923 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:54:12.932 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:56:12.981 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:56:13.016 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:56:14.831 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:56:14.847 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:56:14.847 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:56:14.847 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:56:14.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:56:15.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:56:15.107 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:56:15.332 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:56:15.332 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:56:15.374 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:56:15.401 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:56:15.440 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:56:15.442 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:56:15.478 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:56:16.546 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:58:16.609 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T02:58:16.642 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:58:18.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T02:58:18.369 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T02:58:18.370 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T02:58:18.370 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:58:18.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:58:18.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T02:58:18.612 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T02:58:18.869 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T02:58:18.869 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T02:58:18.899 INFO:teuthology.misc:Compressing logs... 2021-03-21T02:58:18.939 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T02:58:18.966 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T02:58:18.967 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T02:58:19.001 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T02:58:19.941 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:00:19.987 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:00:20.025 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:00:21.112 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:00:21.132 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:00:21.132 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:00:21.132 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:00:21.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:00:21.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:00:21.375 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:00:21.623 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T03:00:21.623 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T03:00:21.663 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:00:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T03:00:21.730 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T03:00:21.731 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T03:00:21.770 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:00:22.820 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:02:22.872 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:02:22.906 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:02:23.881 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:02:23.898 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:02:23.898 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:02:23.898 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:02:23.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:02:24.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:02:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:02:24.384 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T03:02:24.384 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T03:02:24.426 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:02:24.453 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T03:02:24.492 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T03:02:24.494 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T03:02:24.528 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:02:25.521 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:04:25.573 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:04:25.610 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:04:27.709 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:04:27.725 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:04:27.725 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:04:27.725 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:04:27.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:04:27.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:04:27.972 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:04:28.228 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T03:04:28.228 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T03:04:28.264 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:04:28.298 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T03:04:28.330 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T03:04:28.332 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T03:04:28.372 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:04:29.484 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:06:29.531 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:06:29.569 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:06:30.706 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:06:30.724 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:06:30.724 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:06:30.725 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:06:30.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:06:30.939 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:06:30.967 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:06:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T03:06:31.215 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T03:06:31.256 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:06:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T03:06:31.324 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T03:06:31.326 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T03:06:31.390 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:06:32.299 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:08:32.351 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:08:32.384 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:08:34.167 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:08:34.183 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:08:34.184 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:08:34.184 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:08:34.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:08:34.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:08:34.441 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:08:52.810 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:08:52.848 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:08:53.894 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:10:53.947 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:10:54.029 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:10:55.825 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:10:55.841 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:10:55.842 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:10:55.842 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:10:55.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:10:56.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:10:56.093 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:10:59.142 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:10:59.181 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:11:00.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:13:00.975 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:13:01.011 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:13:02.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:13:02.734 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:13:02.734 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:13:02.734 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:13:02.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:13:02.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:13:02.982 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:13:06.018 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:13:06.054 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:13:07.158 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:15:07.213 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:15:07.247 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:15:09.054 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:15:09.071 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:15:09.071 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:15:09.071 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:15:09.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:15:09.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:15:09.332 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:15:12.354 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:15:12.392 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:15:13.926 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:17:13.978 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:17:14.013 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:17:15.111 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:17:15.128 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:17:15.128 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:17:15.128 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:17:15.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:17:15.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:17:15.373 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:17:18.402 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:17:18.438 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:17:19.337 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:19:19.386 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:19:19.420 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:19:20.416 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:19:20.433 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:19:20.433 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:19:20.433 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:19:20.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:19:20.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:19:20.676 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:19:23.714 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:19:23.749 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:19:24.684 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:21:24.733 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:21:24.773 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:21:26.597 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:21:26.613 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:21:26.613 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:21:26.613 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:21:26.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:21:26.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:21:26.872 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:21:29.890 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:21:29.928 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:21:31.633 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:23:31.684 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:23:31.721 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:23:33.485 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:23:33.502 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:23:33.502 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:23:33.502 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:23:33.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:23:33.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:23:33.751 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:23:36.802 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:23:36.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:23:37.907 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:25:37.958 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:25:37.992 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:25:38.926 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:25:38.941 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:25:38.941 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:25:38.941 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:25:38.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:25:39.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:25:39.185 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:25:42.210 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:25:42.243 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:25:43.162 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:27:43.210 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:27:43.246 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:27:44.227 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:27:44.245 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:27:44.245 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:27:44.245 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:27:44.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:27:44.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:27:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:27:47.522 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:27:47.557 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:27:48.523 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:29:48.573 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:29:48.638 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:29:49.611 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:29:49.628 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:29:49.628 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:29:49.628 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:29:49.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:29:49.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:29:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:29:52.902 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:29:52.935 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:29:53.931 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:31:53.984 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:31:54.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:31:54.988 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:31:55.005 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:31:55.005 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:31:55.005 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:31:55.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:31:55.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:31:55.255 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:31:58.306 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:31:58.343 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:31:59.339 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:33:59.395 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:33:59.431 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:34:00.464 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:34:00.481 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:34:00.481 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:34:00.482 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:34:00.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:34:00.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:34:00.738 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:34:03.746 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:34:03.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:34:04.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:36:04.766 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:36:04.803 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:36:05.839 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:36:05.856 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:36:05.856 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:36:05.856 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:36:05.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:36:06.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:36:06.101 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:36:09.122 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:36:09.158 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:36:10.159 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:38:10.207 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:38:10.239 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:38:11.216 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:38:11.231 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:38:11.231 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:38:11.231 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:38:11.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:38:11.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:38:11.476 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:38:14.498 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:38:14.531 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:38:15.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:40:15.518 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:40:15.554 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:40:16.538 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:40:16.553 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:40:16.553 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:40:16.554 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:40:16.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:40:16.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:40:16.812 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:40:19.842 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:40:19.877 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:40:20.809 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:42:20.858 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:42:20.891 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:42:21.838 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:42:21.855 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:42:21.856 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:42:21.856 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:42:21.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:42:22.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:42:22.098 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:42:25.122 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:42:25.155 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:42:26.146 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:44:26.201 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:44:26.235 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:44:27.184 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:44:27.200 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:44:27.200 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:44:27.200 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:44:27.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:44:27.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:44:27.446 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:44:30.498 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:44:30.531 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:44:31.453 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:46:31.507 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:46:31.541 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:46:32.532 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:46:32.555 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:46:32.555 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:46:32.555 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:46:32.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:46:32.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:46:32.814 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:46:35.846 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:46:35.881 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:46:36.825 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:48:36.874 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:48:36.909 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:48:37.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:48:37.917 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:48:37.918 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:48:37.918 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:48:37.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:48:38.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:48:38.161 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:48:41.187 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:48:41.220 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:48:42.198 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:50:42.248 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:50:42.327 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:50:43.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:50:43.255 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:50:43.255 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:50:43.255 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:50:43.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:50:43.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:50:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:50:46.530 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:50:46.563 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:50:47.463 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:52:47.511 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:52:47.544 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:52:48.518 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:52:48.534 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:52:48.534 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:52:48.534 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:52:48.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:52:48.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:52:48.789 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:52:51.811 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:52:51.844 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:52:52.842 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:54:52.890 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:54:52.926 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:54:53.971 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:54:53.988 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:54:53.988 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:54:53.988 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:54:53.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:54:54.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:54:54.238 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:54:57.282 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:54:57.315 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:54:58.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:56:58.322 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:56:58.360 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:56:59.458 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:56:59.474 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:56:59.474 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:56:59.475 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:56:59.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:56:59.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:56:59.720 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:57:02.754 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:57:02.789 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:57:03.769 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:59:03.823 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T03:59:03.859 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:59:04.844 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T03:59:04.861 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T03:59:04.861 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T03:59:04.861 INFO:teuthology.misc:Compressing logs... 2021-03-21T03:59:04.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:59:05.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T03:59:05.120 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T03:59:08.130 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T03:59:08.163 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T03:59:09.094 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:01:09.142 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:01:09.182 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:01:11.179 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:01:11.196 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:01:11.197 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:01:11.197 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:01:11.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:01:11.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:01:11.443 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:01:14.466 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:01:14.504 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:01:15.560 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:03:15.627 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:03:15.661 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:03:17.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:03:17.813 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:03:17.813 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:03:17.813 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:03:17.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:03:18.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:03:18.067 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:03:21.094 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:03:21.130 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:03:22.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:05:22.553 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:05:22.589 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:05:24.079 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:05:24.095 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:05:24.095 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:05:24.095 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:05:24.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:05:24.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:05:24.351 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:05:27.362 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:05:27.398 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:05:28.311 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:07:28.360 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:07:28.394 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:07:29.808 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:07:29.825 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:07:29.825 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:07:29.826 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:07:29.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:07:30.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:07:30.068 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:07:33.090 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:07:33.128 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:07:34.103 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:09:34.162 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:09:34.195 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:09:35.963 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:09:35.979 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:09:35.979 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:09:35.979 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:09:35.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:09:36.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:09:36.223 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:09:39.266 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:09:39.302 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:09:40.293 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:11:40.345 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:11:40.380 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:11:41.559 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:11:41.576 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:11:41.576 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:11:41.576 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:11:41.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:11:41.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:11:41.835 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:11:44.866 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:11:44.902 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:11:45.838 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:13:45.887 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:13:45.920 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:13:46.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:13:46.894 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:13:46.894 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:13:46.895 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:13:46.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:13:47.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:13:47.141 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:13:50.178 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:13:50.213 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:13:51.165 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:15:51.213 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:15:51.247 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:15:52.250 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:15:52.266 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:15:52.266 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:15:52.266 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:15:52.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:15:52.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:15:52.508 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:15:55.554 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:15:55.632 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:15:56.616 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:17:56.666 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:17:56.703 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:17:57.787 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:17:57.806 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:17:57.806 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:17:57.806 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:17:57.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:17:58.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:17:58.061 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:18:01.090 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:18:01.128 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:18:02.085 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:20:02.134 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:20:02.171 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:20:03.377 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:20:03.394 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:20:03.394 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:20:03.395 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:20:03.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:20:03.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:20:03.638 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:20:06.690 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:20:06.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:20:07.698 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:22:07.756 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:22:07.790 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:22:09.115 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:22:09.130 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:22:09.130 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:22:09.130 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:22:09.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:22:09.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:22:09.374 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:22:12.418 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:22:12.454 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:22:13.385 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:24:13.449 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:24:13.482 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:24:14.583 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:24:14.599 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:24:14.599 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:24:14.599 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:24:14.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:24:14.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:24:14.859 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:24:17.890 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:24:17.926 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:24:18.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:26:18.927 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:26:18.960 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:26:20.021 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:26:20.040 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:26:20.040 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:26:20.040 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:26:20.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:26:20.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:26:20.283 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:26:23.330 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:26:23.362 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:26:24.288 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:28:24.339 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:28:24.376 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:28:25.430 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:28:25.446 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:28:25.446 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:28:25.446 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:28:25.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:28:25.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:28:25.688 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:28:28.738 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:28:28.817 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:28:29.732 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:30:29.783 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:30:29.817 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:30:30.876 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:30:30.894 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:30:30.894 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:30:30.894 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:30:30.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:30:31.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:30:31.153 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:30:34.179 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:30:34.212 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:30:35.135 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:32:35.184 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:32:35.216 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:32:36.182 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:32:36.200 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:32:36.200 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:32:36.200 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:32:36.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:32:36.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:32:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:32:39.491 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:32:39.525 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:32:40.584 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:34:40.639 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:34:40.673 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:34:42.290 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:34:42.307 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:34:42.307 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:34:42.307 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:34:42.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:34:42.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:34:42.556 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:34:45.602 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:34:45.639 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:34:46.602 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:36:46.653 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:36:46.687 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:36:47.971 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:36:47.988 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:36:47.988 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:36:47.988 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:36:47.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:36:48.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:36:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:36:51.270 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:36:51.304 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:36:52.221 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:38:52.272 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:38:52.305 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:38:53.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:38:53.316 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:38:53.316 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:38:53.316 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:38:53.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:38:53.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:38:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:38:56.610 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:38:56.646 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:38:57.611 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:40:57.661 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:40:57.694 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:40:58.781 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:40:58.797 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:40:58.797 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:40:58.797 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:40:58.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:40:59.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:40:59.043 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:41:02.082 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:41:02.116 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:41:03.080 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:43:03.131 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:43:03.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:43:04.931 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:43:04.947 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:43:04.947 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:43:04.948 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:43:04.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:43:05.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:43:05.210 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:43:08.226 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:43:08.260 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:43:09.216 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:45:09.270 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:45:09.308 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:45:11.107 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:45:11.124 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:45:11.124 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:45:11.124 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:45:11.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:45:11.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:45:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:45:14.402 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:45:14.437 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:45:15.416 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:47:15.467 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:47:15.504 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:47:16.692 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:47:16.709 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:47:16.709 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:47:16.709 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:47:16.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:47:16.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:47:16.952 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:47:16.980 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T04:47:18.037 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T04:47:19.140 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T04:47:20.236 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T04:47:21.330 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T04:47:22.388 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T04:47:23.485 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T04:47:24.584 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T04:47:25.683 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T04:47:26.781 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T04:47:26.781 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:47:26.817 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:47:27.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:49:27.874 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:49:27.909 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:49:29.674 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:49:29.694 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:49:29.694 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:49:29.694 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:49:29.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:49:29.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:49:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:49:45.926 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:49:45.960 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T04:49:45.960 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:49:46.293 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T04:49:46.293 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:49:46.329 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:49:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T04:49:46.659 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T04:49:46.691 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T04:49:46.691 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:49:46.980 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T04:49:46.980 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:49:46.992 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:49:47.022 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T04:49:47.317 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T04:49:47.351 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T04:49:47.351 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:49:47.379 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T04:49:47.379 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:49:47.416 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:49:49.369 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:51:49.422 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:51:49.460 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:51:51.621 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:51:51.639 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:51:51.639 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:51:51.640 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:51:51.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:51:51.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:51:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:51:52.099 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:51:52.132 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T04:51:52.132 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:51:52.169 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T04:51:52.169 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:51:52.170 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:51:52.202 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T04:51:52.237 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T04:51:52.272 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T04:51:52.272 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:51:52.303 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T04:51:52.304 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:51:52.305 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:51:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T04:51:52.370 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T04:51:52.408 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T04:51:52.408 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:51:52.436 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T04:51:52.436 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:51:52.478 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:51:53.699 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:53:53.754 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:53:53.805 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:53:56.559 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:53:56.580 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:53:56.580 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:53:56.580 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:53:56.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:53:56.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:53:56.832 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:54:15.235 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T04:54:15.290 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:54:18.107 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:56:18.167 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:56:18.242 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:56:21.443 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:56:21.467 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:56:21.467 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:56:21.468 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:56:21.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:56:21.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:56:21.757 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:56:21.956 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:56:21.996 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T04:56:21.996 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:56:22.047 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T04:56:22.047 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:56:22.048 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:56:22.076 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T04:56:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T04:56:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:56:22.218 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:56:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T04:56:22.265 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T04:56:22.316 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T04:56:22.317 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:56:22.342 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T04:56:22.342 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:56:22.399 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:56:24.997 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:58:25.076 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T04:58:25.207 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:58:29.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T04:58:29.991 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T04:58:29.991 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T04:58:29.991 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:58:29.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:58:30.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T04:58:30.268 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:58:30.499 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T04:58:30.531 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T04:58:30.531 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:58:30.568 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T04:58:30.568 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:58:30.575 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:58:30.620 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T04:58:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T04:58:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:58:30.782 INFO:teuthology.misc:Compressing logs... 2021-03-21T04:58:30.860 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T04:58:30.897 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T04:58:30.939 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T04:58:30.940 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:58:30.968 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T04:58:30.969 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T04:58:31.043 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T04:58:33.893 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:00:33.975 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:00:34.025 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:00:35.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:00:35.381 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:00:35.381 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:00:35.381 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:00:35.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:00:35.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:00:35.655 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:00:35.866 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:00:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:00:35.905 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:00:35.940 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:00:35.940 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:00:35.942 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:00:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:00:36.008 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:00:36.012 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:00:36.055 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:00:37.342 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:02:37.395 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:02:37.443 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:02:38.488 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:02:38.504 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:02:38.504 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:02:38.504 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:02:38.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:02:38.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:02:38.765 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:02:38.997 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:02:38.997 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:02:39.036 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:02:39.066 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:02:39.101 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:02:39.102 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:02:39.140 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:02:40.135 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:04:40.184 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:04:40.221 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:04:41.311 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:04:41.328 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:04:41.328 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:04:41.328 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:04:41.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:04:41.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:04:41.571 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:04:41.801 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:04:41.801 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:04:41.840 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:04:41.870 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:04:41.906 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:04:41.908 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:04:41.945 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:04:42.995 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:06:43.044 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:06:43.162 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:06:44.332 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:06:44.349 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:06:44.349 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:06:44.349 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:06:44.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:06:44.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:06:44.599 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:06:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:06:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:06:44.879 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:06:44.908 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:06:44.959 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:06:44.961 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:06:45.000 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:06:46.078 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:08:46.128 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:08:46.166 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:08:47.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:08:47.214 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:08:47.214 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:08:47.214 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:08:47.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:08:47.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:08:47.474 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:08:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:08:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:08:47.831 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:08:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:08:47.859 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:08:47.861 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:08:47.900 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:08:48.975 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:10:49.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:10:49.086 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:10:50.259 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:10:50.276 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:10:50.276 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:10:50.276 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:10:50.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:10:50.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:10:50.529 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:10:50.807 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:10:50.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:10:50.854 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:10:50.883 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:10:50.937 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:10:50.940 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:10:50.976 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:10:52.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:12:52.061 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:12:52.106 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:12:53.288 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:12:53.314 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:12:53.314 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:12:53.315 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:12:53.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:12:53.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:12:53.566 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:12:54.267 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:12:54.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:12:54.801 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:12:54.831 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:12:54.918 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:12:54.923 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:12:54.962 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:12:56.066 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:14:56.123 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:14:56.162 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:14:57.306 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:14:57.340 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:14:57.340 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:14:57.341 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:14:57.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:14:57.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:14:57.607 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:14:57.890 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:14:57.891 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:14:57.938 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:14:57.967 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:14:58.031 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:14:58.033 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:14:58.086 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:14:59.248 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:16:59.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:16:59.336 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:17:00.462 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:17:00.481 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:17:00.481 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:17:00.481 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:17:00.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:17:00.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:17:00.731 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:17:01.063 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:17:01.063 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:17:01.118 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:17:01.146 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:17:01.161 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:17:01.162 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:17:01.202 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:17:02.339 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:19:02.392 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:19:02.433 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:19:03.504 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:19:03.523 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:19:03.523 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:19:03.523 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:19:03.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:19:03.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:19:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:19:04.110 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:19:04.110 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:19:04.157 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:19:04.185 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:19:04.199 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:19:04.201 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:19:04.240 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:19:05.268 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:21:05.320 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:21:05.478 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:21:07.469 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:21:07.496 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:21:07.497 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:21:07.497 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:21:07.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:21:07.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:21:07.768 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:21:08.259 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:21:08.260 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:21:08.703 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:21:08.734 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:21:08.914 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:21:08.916 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:21:08.971 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:21:10.015 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:23:10.069 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:23:10.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:23:11.234 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:23:11.259 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:23:11.261 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:23:11.261 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:23:11.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:23:11.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:23:11.524 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:23:11.832 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:23:11.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:23:11.902 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:23:11.932 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:23:11.952 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:23:11.954 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:23:11.994 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:23:13.124 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:25:13.174 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:25:13.227 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:25:14.270 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:25:14.287 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:25:14.287 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:25:14.287 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:25:14.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:25:14.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:25:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:25:15.308 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:25:15.308 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:25:15.599 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:25:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:25:15.805 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:25:15.808 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:25:15.852 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:25:16.961 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:27:17.017 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:27:17.055 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:27:18.146 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:27:18.163 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:27:18.163 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:27:18.163 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:27:18.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:27:18.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:27:18.425 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:27:19.263 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:27:19.263 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:27:19.707 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:27:19.737 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:27:19.889 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:27:19.892 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:27:19.935 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:27:21.078 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:29:21.132 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:29:21.171 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:29:22.322 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:29:22.342 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:29:22.342 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:29:22.342 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:29:22.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:29:22.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:29:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:29:22.882 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:29:22.882 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:29:22.926 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:29:22.954 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:29:23.003 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:29:23.005 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:29:23.043 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:29:24.119 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:31:24.170 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:31:24.212 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:31:26.200 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:31:26.217 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:31:26.217 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:31:26.217 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:31:26.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:31:26.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:31:26.469 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:31:27.618 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:31:27.619 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:31:28.319 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:31:28.350 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:31:28.492 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:31:28.493 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:31:28.535 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:31:29.608 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:33:29.648 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:33:29.686 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:33:31.684 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:33:31.701 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:33:31.701 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:33:31.701 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:33:31.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:33:31.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:33:31.964 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:33:32.976 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:33:32.976 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:33:33.376 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:33:33.405 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:33:33.477 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:33:33.484 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:33:33.529 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:33:34.591 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:35:34.685 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:35:34.724 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:35:35.797 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:35:35.815 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:35:35.816 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:35:35.816 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:35:35.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:35:36.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:35:36.062 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:35:36.422 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:35:36.423 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:35:36.502 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:35:36.533 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:35:36.573 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:35:36.575 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:35:36.612 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:35:37.690 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:37:37.741 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:37:37.782 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:37:38.870 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:37:38.887 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:37:38.887 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:37:38.888 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:37:38.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:37:39.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:37:39.133 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:37:39.945 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:37:39.945 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:37:40.539 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:37:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:37:40.738 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:37:40.740 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:37:40.781 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:37:41.842 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:39:41.895 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:39:41.934 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:39:43.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:39:43.058 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:39:43.058 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:39:43.058 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:39:43.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:39:43.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:39:43.321 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:39:43.564 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:39:43.564 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:39:43.612 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:39:43.640 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:39:43.688 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:39:43.689 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:39:43.771 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:39:44.774 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:41:44.824 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:41:44.864 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:41:45.916 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:41:45.933 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:41:45.934 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:41:45.934 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:41:45.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:41:46.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:41:46.184 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:41:46.504 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:41:46.505 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:41:46.554 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:41:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:41:46.627 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:41:46.629 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:41:46.674 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:41:47.710 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:43:47.759 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:43:47.799 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:43:48.917 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:43:48.934 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:43:48.934 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:43:48.934 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:43:48.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:43:49.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:43:49.181 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:43:49.460 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:43:49.461 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:43:49.598 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:43:49.628 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:43:49.633 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:43:49.634 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:43:49.673 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:43:50.776 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:45:50.830 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:45:50.869 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:45:52.004 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:45:52.023 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:45:52.023 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:45:52.023 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:45:52.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:45:52.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:45:52.288 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:45:52.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:45:52.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:45:52.680 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:45:52.707 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:45:52.709 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:45:52.712 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:45:52.755 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:45:53.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:47:53.930 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:47:53.970 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:47:55.052 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:47:55.085 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:47:55.086 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:47:55.086 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:47:55.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:47:55.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:47:55.332 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:47:55.597 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:47:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:47:55.643 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:47:55.673 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:47:55.719 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:47:55.721 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:47:55.762 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:47:56.862 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:49:56.925 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:49:56.976 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:49:58.827 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:49:58.845 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:49:58.845 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:49:58.845 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:49:58.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:49:59.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:49:59.090 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:49:59.594 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:49:59.595 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:49:59.766 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:49:59.799 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:49:59.876 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:49:59.877 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:49:59.914 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:50:00.978 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:52:01.029 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:52:01.181 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:52:03.223 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:52:03.241 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:52:03.241 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:52:03.241 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:52:03.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:52:03.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:52:03.508 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:52:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:52:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:52:03.815 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:52:03.844 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:52:03.882 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:52:03.884 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:52:03.926 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:52:05.053 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:54:05.104 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:54:05.142 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:54:07.005 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:54:07.021 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:54:07.021 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:54:07.021 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:54:07.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:54:07.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:54:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:54:07.521 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:54:07.521 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:54:07.610 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:54:07.647 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:54:07.653 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:54:07.655 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:54:07.715 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:54:08.822 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:56:08.873 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:56:08.913 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:56:10.760 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:56:10.779 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:56:10.779 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:56:10.779 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:56:10.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:56:10.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:56:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:56:11.279 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:56:11.279 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:56:11.357 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:56:11.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:56:11.397 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:56:11.398 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:56:11.436 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:56:12.552 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:58:12.613 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T05:58:12.653 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:58:14.498 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T05:58:14.515 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T05:58:14.515 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T05:58:14.515 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:58:14.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:58:14.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T05:58:14.783 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T05:58:15.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T05:58:15.391 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T05:58:15.454 INFO:teuthology.misc:Compressing logs... 2021-03-21T05:58:15.483 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T05:58:15.489 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T05:58:15.490 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T05:58:15.533 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T05:58:16.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:00:16.694 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:00:16.737 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:00:17.898 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:00:17.915 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:00:17.915 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:00:17.915 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:00:17.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:00:18.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:00:18.161 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:00:18.405 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:00:18.405 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:00:18.456 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:00:18.484 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:00:18.522 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:00:18.523 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:00:18.562 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:00:19.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:02:19.666 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:02:19.706 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:02:20.802 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:02:20.819 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:02:20.819 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:02:20.819 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:02:20.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:02:21.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:02:21.065 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:02:21.294 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:02:21.295 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:02:21.332 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:02:21.365 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:02:21.398 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:02:21.399 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:02:21.437 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:02:22.531 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:04:22.586 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:04:22.624 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:04:23.772 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:04:23.792 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:04:23.792 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:04:23.792 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:04:23.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:04:24.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:04:24.054 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:04:24.293 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:04:24.293 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:04:24.321 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:04:24.363 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:04:24.386 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:04:24.388 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:04:24.429 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:04:25.672 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:06:25.726 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:06:25.764 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:06:26.873 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:06:26.891 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:06:26.891 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:06:26.891 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:06:26.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:06:27.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:06:27.141 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:06:27.400 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:06:27.400 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:06:27.479 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:06:27.507 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:06:27.509 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:06:27.512 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:06:27.549 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:06:28.718 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:08:28.767 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:08:28.884 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:08:29.939 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:08:29.960 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:08:29.960 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:08:29.961 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:08:29.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:08:30.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:08:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:08:30.657 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:08:30.658 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:08:30.717 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:08:30.785 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:08:30.787 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:08:30.788 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:08:30.849 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:08:32.003 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:10:32.057 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:10:32.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:10:33.161 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:10:33.178 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:10:33.178 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:10:33.178 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:10:33.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:10:33.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:10:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:10:33.700 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:10:33.700 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:10:33.781 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:10:33.808 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:10:33.809 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:10:33.811 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:10:33.850 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:10:34.907 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:12:34.957 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:12:34.997 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:12:36.161 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:12:36.178 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:12:36.179 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:12:36.179 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:12:36.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:12:36.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:12:36.423 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:12:36.683 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:12:36.683 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:12:36.753 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:12:36.781 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:12:36.783 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:12:36.786 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:12:36.826 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:12:37.923 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:14:37.977 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:14:38.048 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:14:39.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:14:39.288 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:14:39.289 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:14:39.289 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:14:39.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:14:39.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:14:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:14:39.827 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:14:39.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:14:39.902 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:14:39.934 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:14:39.934 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:14:39.937 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:14:39.977 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:14:41.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:16:41.096 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:16:41.138 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:16:42.278 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:16:42.295 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:16:42.295 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:16:42.295 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:16:42.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:16:42.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:16:42.561 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:16:42.791 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:16:42.791 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:16:42.820 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:16:42.860 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:16:42.885 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:16:42.887 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:16:42.927 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:16:43.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:18:44.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:18:44.094 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:18:45.142 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:18:45.160 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:18:45.160 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:18:45.160 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:18:45.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:18:45.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:18:45.408 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:18:45.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:18:45.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:18:45.729 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:18:45.758 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:18:45.795 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:18:45.796 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:18:45.834 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:18:46.864 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:20:46.914 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:20:46.956 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:20:48.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:20:48.089 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:20:48.089 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:20:48.089 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:20:48.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:20:48.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:20:48.339 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:20:48.646 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:20:48.646 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:20:48.693 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:20:48.723 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:20:48.736 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:20:48.737 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:20:48.783 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:20:49.921 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:22:49.974 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:22:50.012 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:22:51.151 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:22:51.168 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:22:51.168 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:22:51.168 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:22:51.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:22:51.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:22:51.429 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:22:51.695 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:22:51.695 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:22:51.749 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:22:51.779 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:22:51.782 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:22:51.782 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:22:51.824 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:22:52.925 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:24:52.988 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:24:53.030 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:24:54.099 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:24:54.120 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:24:54.120 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:24:54.121 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:24:54.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:24:54.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:24:54.365 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:24:54.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:24:54.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:24:54.713 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:24:54.741 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:24:54.742 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:24:54.745 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:24:54.782 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:24:55.924 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:26:55.991 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:26:56.050 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:26:57.312 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:26:57.336 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:26:57.336 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:26:57.336 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:26:57.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:26:57.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:26:57.589 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:26:57.898 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:26:57.898 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:26:57.944 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:26:57.975 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:26:58.009 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:26:58.010 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:26:58.055 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:26:59.218 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:28:59.280 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:28:59.325 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:29:00.515 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:29:00.532 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:29:00.532 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:29:00.533 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:29:00.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:29:00.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:29:00.795 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:29:01.044 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:29:01.044 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:29:01.099 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:29:01.127 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:29:01.127 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:29:01.130 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:29:01.194 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:29:02.306 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:31:02.360 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:31:02.401 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:31:03.587 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:31:03.609 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:31:03.610 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:31:03.610 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:31:03.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:31:03.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:31:03.861 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:31:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:31:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:31:04.132 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:31:04.163 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:31:04.197 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:31:04.199 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:31:04.242 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:31:05.387 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:33:05.441 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:33:05.481 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:33:06.564 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:33:06.583 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:33:06.584 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:33:06.584 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:33:06.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:33:06.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:33:06.834 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:33:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:33:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:33:07.195 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:33:07.230 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:33:07.230 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:33:07.232 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:33:07.276 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:33:08.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:35:08.430 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:35:08.478 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:35:10.681 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:35:10.699 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:35:10.699 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:35:10.699 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:35:10.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:35:10.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:35:10.969 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:35:11.314 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:35:11.315 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:35:11.403 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:35:11.448 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:35:11.466 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:35:11.470 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:35:11.528 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:35:14.273 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:37:14.398 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:37:14.439 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:37:16.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:37:16.872 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:37:16.872 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:37:16.872 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:37:16.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:37:17.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:37:17.121 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:37:17.420 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:37:17.421 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:37:17.472 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:37:17.501 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:37:17.502 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:37:17.504 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:37:17.603 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:37:19.450 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:39:19.513 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:39:19.556 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:39:21.428 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:39:21.452 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:39:21.453 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:39:21.453 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:39:21.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:39:21.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:39:21.712 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:39:21.955 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:39:21.955 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:39:21.998 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:39:22.027 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:39:22.066 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:39:22.069 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:39:22.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:39:23.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:41:23.332 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:41:23.392 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:41:24.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:41:24.654 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:41:24.654 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:41:24.654 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:41:24.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:41:24.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:41:24.917 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:41:25.159 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:41:25.159 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:41:25.191 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:41:25.228 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:41:25.258 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:41:25.260 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:41:25.300 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:41:26.509 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:43:26.563 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:43:26.602 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:43:28.596 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:43:28.614 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:43:28.614 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:43:28.614 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:43:28.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:43:28.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:43:28.870 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:43:29.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:43:29.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:43:29.152 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:43:29.181 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:43:29.221 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:43:29.223 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:43:29.261 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:43:30.419 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:45:30.474 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:45:30.516 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:45:32.663 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:45:32.688 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:45:32.689 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:45:32.689 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:45:32.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:45:32.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:45:32.936 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:45:33.173 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:45:33.173 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:45:33.212 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:45:33.244 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:45:33.278 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:45:33.279 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:45:33.321 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:45:34.526 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:47:34.579 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:47:34.665 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:47:35.991 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:47:36.010 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:47:36.010 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:47:36.010 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:47:36.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:47:36.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:47:36.277 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:47:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:47:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:47:36.557 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:47:36.588 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:47:36.625 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:47:36.626 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:47:36.677 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:47:37.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:49:37.854 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:49:37.953 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:49:39.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:49:39.955 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:49:39.955 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:49:39.955 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:49:39.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:49:40.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:49:40.208 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:49:40.449 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:49:40.449 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:49:40.490 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:49:40.519 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:49:40.557 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:49:40.558 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:49:40.597 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:49:41.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:51:41.744 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:51:41.791 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:51:43.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:51:43.737 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:51:43.738 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:51:43.738 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:51:43.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:51:43.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:51:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:51:44.234 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:51:44.234 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:51:44.272 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:51:44.304 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:51:44.340 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:51:44.341 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:51:44.381 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:51:45.482 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:53:45.542 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:53:45.590 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:53:47.722 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:53:47.741 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:53:47.742 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:53:47.742 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:53:47.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:53:47.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:53:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:53:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:53:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:53:48.292 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:53:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:53:48.359 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:53:48.361 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:53:48.404 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:53:49.984 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:55:50.040 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:55:50.086 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:55:52.099 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:55:52.120 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:55:52.121 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:55:52.121 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:55:52.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:55:52.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:55:52.366 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:55:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:55:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:55:52.648 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:55:52.678 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:55:52.715 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:55:52.717 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:55:52.756 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:55:54.184 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:57:54.233 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:57:54.270 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:57:56.148 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:57:56.167 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T06:57:56.167 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T06:57:56.167 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:57:56.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:57:56.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T06:57:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T06:57:56.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T06:57:56.741 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T06:57:56.776 INFO:teuthology.misc:Compressing logs... 2021-03-21T06:57:56.819 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T06:57:56.841 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T06:57:56.843 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T06:57:56.882 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T06:57:58.003 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T06:59:58.069 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T06:59:58.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:00:00.304 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:00:00.327 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:00:00.327 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:00:00.327 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:00:00.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:00:00.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:00:00.594 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:00:00.821 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:00:00.821 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:00:00.860 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:00:00.889 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:00:00.926 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:00:00.928 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:00:00.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:00:02.906 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:02:02.964 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:02:03.028 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:02:05.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:02:05.186 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:02:05.186 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:02:05.186 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:02:05.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:02:05.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:02:05.437 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:02:05.666 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:02:05.666 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:02:05.704 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:02:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:02:05.768 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:02:05.769 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:02:05.809 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:02:06.960 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:04:07.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:04:07.092 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:04:08.983 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:04:09.017 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:04:09.017 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:04:09.017 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:04:09.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:04:09.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:04:09.277 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:04:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:04:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:04:09.544 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:04:09.576 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:04:09.610 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:04:09.611 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:04:09.650 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:04:10.787 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:06:10.835 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:06:10.878 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:06:12.714 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:06:12.731 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:06:12.731 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:06:12.731 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:06:12.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:06:12.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:06:12.994 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:06:13.221 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:06:13.222 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:06:13.259 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:06:13.290 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:06:13.325 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:06:13.326 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:06:13.366 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:06:14.446 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:08:14.497 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:08:14.538 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:08:16.332 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:08:16.349 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:08:16.349 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:08:16.350 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:08:16.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:08:16.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:08:16.595 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:08:16.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:08:16.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:08:16.867 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:08:16.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:08:16.933 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:08:16.934 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:08:16.973 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:08:18.008 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:10:18.056 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:10:18.108 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:10:20.048 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:10:20.071 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:10:20.071 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:10:20.072 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:10:20.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:10:20.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:10:20.316 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:10:20.551 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:10:20.552 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:10:20.589 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:10:20.621 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:10:20.654 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:10:20.656 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:10:20.700 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:10:21.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:12:21.864 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:12:21.905 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:12:23.006 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:12:23.026 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:12:23.026 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:12:23.026 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:12:23.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:12:23.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:12:23.293 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:12:23.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:12:23.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:12:23.550 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:12:23.590 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:12:23.616 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:12:23.617 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:12:23.666 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:12:24.836 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:14:24.889 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:14:24.935 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:14:27.459 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:14:27.476 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:14:27.477 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:14:27.477 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:14:27.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:14:27.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:14:27.729 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:14:46.083 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T07:14:46.127 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:14:48.301 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:16:48.358 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:16:48.397 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:16:50.433 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:16:50.451 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:16:50.451 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:16:50.452 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:16:50.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:16:50.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:16:50.704 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:17:50.678 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T07:17:50.723 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:17:52.695 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:19:52.741 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:19:52.782 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:19:53.990 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:19:54.007 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:19:54.007 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:19:54.007 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:19:54.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:19:54.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:19:54.271 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:19:54.449 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:19:54.481 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:19:54.481 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:19:54.529 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:19:54.529 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:19:54.530 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:19:54.558 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:19:54.627 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T07:19:54.627 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:19:54.675 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:19:54.701 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T07:19:54.728 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T07:19:54.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T07:19:54.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:19:54.792 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T07:19:54.792 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:19:54.836 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:19:55.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:21:56.050 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:21:56.092 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:21:57.184 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:21:57.204 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:21:57.204 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:21:57.205 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:21:57.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:21:57.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:21:57.456 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:21:57.658 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:21:57.690 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:21:57.690 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:21:57.723 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:21:57.724 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:21:57.725 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:21:57.760 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:21:57.830 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T07:21:57.830 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:21:57.873 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:21:57.901 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T07:21:57.923 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T07:21:57.970 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T07:21:57.970 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:21:57.996 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T07:21:57.996 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:21:58.033 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:21:59.107 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:23:59.189 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:23:59.227 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:24:00.316 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:24:00.334 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:24:00.334 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:24:00.334 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:24:00.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:24:00.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:24:00.579 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:24:00.779 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:24:00.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:24:00.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:24:00.845 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:24:00.845 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:24:00.846 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:24:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:24:00.912 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:24:00.914 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:24:01.025 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:24:02.080 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:26:02.132 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:26:02.172 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:26:03.299 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:26:03.317 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:26:03.317 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:26:03.317 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:26:03.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:26:03.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:26:03.579 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:26:03.796 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:26:03.797 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:26:03.836 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:26:03.866 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:26:03.902 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:26:03.903 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:26:03.941 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:26:05.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:28:05.099 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:28:05.140 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:28:06.189 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:28:06.206 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:28:06.206 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:28:06.206 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:28:06.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:28:06.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:28:06.450 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:28:06.679 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:28:06.679 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:28:06.716 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:28:06.750 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:28:06.780 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:28:06.781 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:28:06.819 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:28:07.871 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:30:07.923 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:30:07.964 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:30:08.999 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:30:09.016 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:30:09.016 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:30:09.016 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:30:09.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:30:09.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:30:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:30:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:30:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:30:09.544 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:30:09.571 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:30:09.609 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:30:09.611 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:30:09.647 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:30:10.718 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:32:10.773 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:32:10.810 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:32:11.870 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:32:11.886 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:32:11.886 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:32:11.887 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:32:11.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:32:12.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:32:12.150 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:32:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:32:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:32:12.394 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:32:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:32:12.459 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:32:12.461 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:32:12.498 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:32:13.516 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:34:13.567 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:34:13.605 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:34:14.680 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:34:14.698 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:34:14.698 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:34:14.698 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:34:14.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:34:14.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:34:14.946 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:34:15.180 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:34:15.180 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:34:15.216 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:34:15.251 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:34:15.280 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:34:15.281 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:34:15.319 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:34:16.370 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:36:16.418 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:36:16.457 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:36:17.525 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:36:17.542 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:36:17.543 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:36:17.543 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:36:17.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:36:17.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:36:17.789 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:36:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:36:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:36:18.061 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:36:18.092 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:36:18.126 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:36:18.128 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:36:18.170 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:36:19.226 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:38:19.279 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:38:19.320 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:38:20.405 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:38:20.422 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:38:20.422 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:38:20.422 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:38:20.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:38:20.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:38:20.687 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:38:20.908 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:38:20.908 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:38:20.948 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:38:20.978 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:38:21.013 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:38:21.016 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:38:21.054 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:38:22.159 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:40:22.211 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:40:22.250 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:40:23.381 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:40:23.400 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:40:23.400 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:40:23.400 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:40:23.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:40:23.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:40:23.646 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:40:23.876 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:40:23.876 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:40:23.914 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:40:23.947 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:40:23.979 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:40:23.981 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:40:24.019 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:40:25.127 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:42:25.178 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:42:25.216 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:42:26.299 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:42:26.316 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:42:26.316 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:42:26.316 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:42:26.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:42:26.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:42:26.559 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:42:26.788 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:42:26.789 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:42:26.829 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:42:26.859 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:42:26.894 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:42:26.895 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:42:26.936 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:42:28.039 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:44:28.100 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:44:28.140 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:44:29.271 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:44:29.288 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:44:29.289 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:44:29.289 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:44:29.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:44:29.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:44:29.557 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:44:29.786 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:44:29.786 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:44:29.827 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:44:29.855 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:44:29.893 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:44:29.894 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:44:29.937 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:44:31.058 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:46:31.109 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:46:31.149 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:46:32.192 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:46:32.209 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:46:32.210 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:46:32.210 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:46:32.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:46:32.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:46:32.457 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:46:32.684 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:46:32.684 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:46:32.722 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:46:32.755 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:46:32.786 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:46:32.788 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:46:32.826 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:46:33.883 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:48:33.935 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:48:33.973 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:48:35.041 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:48:35.069 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:48:35.069 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:48:35.069 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:48:35.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:48:35.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:48:35.321 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:48:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:48:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:48:35.601 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:48:35.643 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:48:35.667 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:48:35.669 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:48:35.707 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:48:36.792 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:50:36.844 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:50:36.886 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:50:37.987 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:50:38.013 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:50:38.013 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:50:38.013 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:50:38.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:50:38.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:50:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:50:56.263 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T07:50:56.301 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:50:57.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:52:57.403 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:52:57.445 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:52:58.576 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:52:58.599 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:52:58.599 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:52:58.599 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:52:58.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:52:58.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:52:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:52:58.881 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T07:52:59.939 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T07:53:01.040 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T07:53:02.099 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T07:53:03.171 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T07:53:04.293 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T07:53:05.390 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T07:53:06.609 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T07:53:07.721 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T07:53:08.819 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T07:53:08.819 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T07:53:08.868 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:53:09.993 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:55:10.056 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:55:10.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:55:12.176 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:55:12.197 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:55:12.197 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:55:12.198 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:55:12.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:55:12.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:55:12.455 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:55:28.590 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:55:28.627 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:55:28.627 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:55:28.668 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:55:28.669 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:55:28.670 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:55:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:55:28.734 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:55:28.773 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T07:55:28.775 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:55:28.798 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T07:55:28.799 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:55:28.800 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:55:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T07:55:28.865 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T07:55:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T07:55:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:55:28.938 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T07:55:28.938 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:55:28.983 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:55:30.134 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:57:30.193 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:57:30.230 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:57:31.296 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:57:31.314 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:57:31.314 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:57:31.314 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:57:31.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:57:31.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:57:31.581 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:57:31.948 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:57:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:57:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:57:32.029 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:57:32.029 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:57:32.030 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:57:32.061 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:57:32.133 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T07:57:32.133 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:57:32.189 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:57:32.218 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T07:57:32.238 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T07:57:32.290 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T07:57:32.290 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:57:32.315 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T07:57:32.315 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:57:32.359 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:57:33.472 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:59:33.526 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T07:59:33.629 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:59:34.747 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T07:59:34.764 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T07:59:34.764 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T07:59:34.764 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:59:34.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:59:34.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T07:59:35.012 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:59:35.216 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T07:59:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:59:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:59:35.281 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T07:59:35.281 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T07:59:35.282 INFO:teuthology.misc:Compressing logs... 2021-03-21T07:59:35.317 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T07:59:35.348 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T07:59:35.350 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T07:59:35.389 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T07:59:36.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:01:36.572 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:01:36.612 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:01:37.748 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:01:37.765 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:01:37.765 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:01:37.765 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:01:37.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:01:37.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:01:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:01:38.205 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:01:38.234 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:01:38.234 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:01:38.270 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:01:38.270 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:01:38.271 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:01:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:01:38.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:01:38.337 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:01:38.375 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:01:39.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:03:39.484 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:03:39.565 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:03:41.659 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:03:41.678 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:03:41.678 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:03:41.678 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:03:41.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:03:41.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:03:41.942 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:03:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:03:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:03:42.206 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:03:42.235 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:03:42.272 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:03:42.274 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:03:42.319 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:03:43.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:05:43.454 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:05:43.495 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:05:44.676 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:05:44.693 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:05:44.693 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:05:44.694 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:05:44.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:05:44.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:05:44.946 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:05:45.186 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:05:45.186 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:05:45.226 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:05:45.256 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:05:45.292 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:05:45.507 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/ceph_test_objectstore.log: file size changed while zipping 2021-03-21T08:05:45.512 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:05:45.550 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:05:46.696 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:07:46.755 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:07:46.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:07:47.928 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:07:47.945 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:07:47.945 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:07:47.945 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:07:47.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:07:48.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:07:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:07:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:07:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:07:48.463 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:07:48.495 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:07:48.528 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:07:48.530 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:07:48.576 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:07:49.647 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:09:49.707 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:09:49.745 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:09:50.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:09:50.869 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:09:50.869 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:09:50.869 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:09:50.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:09:51.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:09:51.129 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:09:51.370 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:09:51.370 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:09:51.399 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:09:51.439 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:09:51.465 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:09:51.466 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:09:51.505 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:09:52.561 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:11:52.616 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:11:52.656 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:11:53.731 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:11:53.748 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:11:53.748 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:11:53.748 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:11:53.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:11:53.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:11:53.992 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:11:54.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:11:54.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:11:54.278 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:11:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:11:54.346 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:11:54.348 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:11:54.387 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:11:55.455 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:13:55.505 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:13:55.542 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:13:56.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:13:56.656 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:13:56.656 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:13:56.657 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:13:56.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:13:56.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:13:56.914 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:13:57.150 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:13:57.150 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:13:57.192 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:13:57.220 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:13:57.260 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:13:57.262 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:13:57.300 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:13:58.337 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:15:58.387 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:15:58.426 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:16:00.226 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:16:00.244 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:16:00.244 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:16:00.244 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:16:00.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:16:00.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:16:00.508 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:16:00.738 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:16:00.738 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:16:00.779 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:16:00.806 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:16:00.843 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:16:00.845 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:16:00.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:16:01.977 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:18:02.027 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:18:02.068 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:18:03.192 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:18:03.210 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:18:03.210 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:18:03.210 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:18:03.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:18:03.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:18:03.458 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:18:03.698 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:18:03.698 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:18:03.734 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:18:03.770 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:18:03.802 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:18:03.804 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:18:03.843 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:18:04.930 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:20:04.978 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:20:05.037 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:20:07.116 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:20:07.135 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:20:07.135 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:20:07.136 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:20:07.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:20:07.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:20:07.459 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:20:07.759 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:20:07.760 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:20:07.794 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:20:07.833 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:20:07.864 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:20:07.866 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:20:07.911 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:20:09.013 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:22:09.072 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:22:09.111 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:22:11.434 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:22:11.465 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:22:11.466 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:22:11.466 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:22:11.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:22:11.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:22:11.729 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:22:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:22:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:22:12.005 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:22:12.043 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:22:12.076 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:22:12.078 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:22:12.120 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:22:14.277 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:24:14.335 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:24:14.406 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:24:16.628 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:24:16.645 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:24:16.646 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:24:16.646 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:24:16.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:24:16.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:24:16.897 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:24:17.147 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:24:17.148 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:24:17.189 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:24:17.218 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:24:17.266 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:24:17.266 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:24:17.307 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:24:18.994 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:26:19.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:26:19.083 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:26:21.123 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:26:21.141 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:26:21.141 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:26:21.141 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:26:21.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:26:21.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:26:21.389 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:26:21.626 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:26:21.626 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:26:21.666 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:26:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:26:21.744 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:26:21.745 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:26:21.785 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:26:23.894 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:28:23.949 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:28:23.990 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:28:25.758 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:28:25.776 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:28:25.776 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:28:25.777 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:28:25.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:28:25.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:28:26.037 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:28:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:28:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:28:26.306 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:28:26.352 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:28:26.390 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:28:26.391 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:28:26.430 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:28:27.531 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:30:27.582 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:30:27.621 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:30:28.697 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:30:28.715 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:30:28.715 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:30:28.715 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:30:28.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:30:28.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:30:28.964 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:30:29.195 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:30:29.195 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:30:29.232 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:30:29.265 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:30:29.315 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:30:29.317 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:30:29.354 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:30:30.415 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:32:30.465 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:32:30.504 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:32:31.520 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:32:31.540 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:32:31.540 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:32:31.540 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:32:31.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:32:31.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:32:31.793 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:32:32.033 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:32:32.034 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:32:32.073 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:32:32.104 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:32:32.140 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:32:32.142 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:32:32.242 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:32:33.262 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:34:33.358 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:34:33.397 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:34:35.240 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:34:35.262 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:34:35.262 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:34:35.262 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:34:35.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:34:35.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:34:35.527 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:34:35.770 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:34:35.770 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:34:35.802 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:34:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:34:35.868 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:34:35.869 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:34:35.910 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:34:37.668 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:36:37.718 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:36:37.772 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:36:39.753 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:36:39.771 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:36:39.771 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:36:39.771 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:36:39.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:36:39.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:36:40.019 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:36:40.256 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:36:40.256 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:36:40.296 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:36:40.326 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:36:40.362 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:36:40.363 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:36:40.403 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:36:41.530 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:38:41.584 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:38:41.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:38:42.795 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:38:42.813 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:38:42.813 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:38:42.814 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:38:42.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:38:43.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:38:43.066 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:38:43.304 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:38:43.304 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:38:43.342 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:38:43.375 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:38:43.410 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:38:43.411 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:38:43.454 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:38:44.565 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:40:44.716 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:40:44.755 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:40:45.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:40:45.832 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:40:45.833 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:40:45.833 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:40:45.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:40:46.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:40:46.098 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:40:46.345 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:40:46.345 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:40:46.376 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:40:46.413 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:40:46.441 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:40:46.444 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:40:46.481 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:40:47.542 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:42:47.597 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:42:47.639 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:42:48.736 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:42:48.753 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:42:48.753 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:42:48.753 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:42:48.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:42:48.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:42:48.999 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:42:49.241 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:42:49.242 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:42:49.281 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:42:49.309 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:42:49.347 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:42:49.349 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:42:49.387 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:42:50.443 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:44:50.495 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:44:50.533 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:44:52.685 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:44:52.702 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:44:52.702 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:44:52.703 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:44:52.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:44:52.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:44:52.949 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:44:53.195 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:44:53.195 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:44:53.233 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:44:53.265 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:44:53.298 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:44:53.300 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:44:53.340 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:44:54.404 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:46:54.455 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:46:54.499 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:46:56.525 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:46:56.545 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:46:56.545 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:46:56.546 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:46:56.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:46:56.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:46:56.813 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:46:57.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:46:57.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:46:57.093 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:46:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:46:57.161 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:46:57.161 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:46:57.200 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:46:58.317 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:48:58.390 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:48:58.435 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:48:59.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:48:59.560 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:48:59.561 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:48:59.561 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:48:59.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:48:59.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:48:59.808 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:49:00.043 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:49:00.043 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:49:00.083 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:49:00.112 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:49:00.149 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:49:00.150 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:49:00.192 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:49:01.235 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:51:01.288 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:51:01.332 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:51:02.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:51:02.518 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:51:02.518 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:51:02.518 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:51:02.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:51:02.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:51:02.846 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:51:03.083 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:51:03.083 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:51:03.122 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:51:03.154 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:51:03.189 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:51:03.190 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:51:03.234 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:51:04.381 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:53:04.435 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:53:04.474 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:53:05.591 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:53:05.608 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:53:05.608 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:53:05.608 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:53:05.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:53:05.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:53:05.868 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:53:06.105 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:53:06.105 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:53:06.134 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:53:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:53:06.200 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:53:06.201 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:53:06.238 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:53:07.272 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:55:07.326 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:55:07.368 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:55:08.404 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:55:08.420 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:55:08.420 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:55:08.421 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:55:08.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:55:08.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:55:08.666 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:55:08.911 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:55:08.911 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:55:08.951 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:55:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:55:09.019 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:55:09.021 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:55:09.060 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:55:10.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:57:10.180 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:57:10.219 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:57:11.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:57:11.368 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:57:11.368 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:57:11.368 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:57:11.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:57:11.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:57:11.620 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:57:11.857 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:57:11.857 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:57:11.895 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:57:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:57:11.960 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:57:11.962 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:57:12.003 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:57:13.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:59:13.123 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T08:59:13.160 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:59:14.202 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T08:59:14.218 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T08:59:14.218 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T08:59:14.218 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:59:14.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:59:14.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T08:59:14.481 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T08:59:14.726 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T08:59:14.726 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T08:59:14.765 INFO:teuthology.misc:Compressing logs... 2021-03-21T08:59:14.797 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T08:59:14.830 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T08:59:14.831 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T08:59:14.869 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T08:59:15.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:01:16.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:01:16.086 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:01:17.250 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:01:17.272 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:01:17.272 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:01:17.273 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:01:17.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:01:17.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:01:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:01:17.762 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:01:17.762 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:01:17.800 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:01:17.830 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:01:17.865 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:01:17.867 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:01:17.911 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:01:19.088 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:03:19.141 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:03:19.254 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:03:21.370 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:03:21.398 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:03:21.398 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:03:21.398 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:03:21.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:03:21.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:03:21.660 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:03:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:03:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:03:21.943 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:03:21.977 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:03:22.008 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:03:22.010 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:03:22.072 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:03:24.097 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:05:24.159 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:05:24.210 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:05:26.289 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:05:26.305 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:05:26.306 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:05:26.306 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:05:26.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:05:26.529 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:05:26.576 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:05:26.821 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:05:26.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:05:26.852 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:05:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:05:26.921 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:05:26.921 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:05:26.961 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:05:28.098 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:07:28.150 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:07:28.204 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:07:29.295 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:07:29.366 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:07:29.366 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:07:29.366 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:07:29.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:07:29.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:07:29.610 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:07:29.845 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:07:29.845 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:07:29.884 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:07:29.914 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:07:29.950 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:07:29.952 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:07:29.990 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:07:31.118 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:09:31.167 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:09:31.209 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:09:33.483 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:09:33.500 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:09:33.501 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:09:33.501 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:09:33.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:09:33.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:09:33.816 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:09:34.068 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:09:34.068 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:09:34.106 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:09:34.136 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:09:34.174 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:09:34.176 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:09:34.224 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:09:36.154 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:11:36.206 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:11:36.268 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:11:38.504 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:11:38.527 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:11:38.527 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:11:38.527 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:11:38.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:11:38.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:11:38.794 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:11:39.034 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:11:39.034 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:11:39.064 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:11:39.104 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:11:39.129 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:11:39.131 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:11:39.171 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:11:41.020 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:13:41.069 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:13:41.111 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:13:43.495 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:13:43.515 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:13:43.515 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:13:43.515 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:13:43.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:13:43.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:13:43.759 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:13:43.995 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:13:43.995 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:13:44.036 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:13:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:13:44.102 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:13:44.103 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:13:44.141 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:13:46.085 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:15:46.146 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:15:46.214 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:15:48.388 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:15:48.420 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:15:48.421 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:15:48.421 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:15:48.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:15:48.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:15:48.678 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:15:48.916 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:15:48.917 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:15:48.955 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:15:48.987 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:15:49.022 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:15:49.022 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:15:49.068 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:15:50.167 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:17:50.219 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:17:50.259 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:17:51.377 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:17:51.424 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:17:51.424 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:17:51.424 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:17:51.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:17:51.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:17:51.772 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:17:52.089 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:17:52.090 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:17:52.128 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:17:52.169 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:17:52.203 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:17:52.205 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:17:52.243 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:17:53.530 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:19:53.581 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:19:53.621 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:19:55.631 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:19:55.651 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:19:55.651 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:19:55.651 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:19:55.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:19:55.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:19:55.900 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:19:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:19:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:19:56.180 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:19:56.214 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:19:56.247 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:19:56.248 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:19:56.286 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:19:58.188 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:21:58.238 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:21:58.276 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:22:00.233 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:22:00.250 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:22:00.251 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:22:00.251 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:22:00.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:22:00.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:22:00.498 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:22:00.737 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:22:00.737 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:22:00.776 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:22:00.809 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:22:00.842 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:22:00.844 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:22:00.885 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:22:01.934 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:24:01.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:24:02.035 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:24:03.117 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:24:03.135 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:24:03.136 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:24:03.136 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:24:03.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:24:03.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:24:03.397 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:24:03.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:24:03.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:24:03.675 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:24:03.704 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:24:03.739 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:24:03.741 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:24:03.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:24:04.859 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:26:04.910 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:26:04.948 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:26:07.098 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:26:07.117 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:26:07.117 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:26:07.117 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:26:07.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:26:07.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:26:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:26:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:26:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:26:07.641 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:26:07.673 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:26:07.707 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:26:07.710 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:26:07.757 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:26:08.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:28:08.905 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:28:08.946 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:28:10.012 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:28:10.030 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:28:10.030 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:28:10.031 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:28:10.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:28:10.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:28:10.277 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:28:10.519 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:28:10.519 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:28:10.556 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:28:10.590 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:28:10.622 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:28:10.623 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:28:10.664 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:28:11.769 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:30:11.827 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:30:11.872 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:30:12.964 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:30:12.981 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:30:12.982 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:30:12.982 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:30:12.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:30:13.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:30:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:30:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:30:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:30:13.531 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:30:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:30:13.596 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:30:13.598 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:30:13.635 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:30:14.680 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:32:14.731 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:32:14.771 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:32:15.854 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:32:15.873 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:32:15.873 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:32:15.874 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:32:15.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:32:16.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:32:16.118 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:32:16.346 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:32:16.346 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:32:16.384 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:32:16.417 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:32:16.449 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:32:16.450 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:32:16.491 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:32:17.614 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:34:17.669 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:34:17.706 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:34:18.756 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:34:18.773 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:34:18.774 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:34:18.774 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:34:18.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:34:18.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:34:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:34:19.259 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:34:19.260 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:34:19.299 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:34:19.331 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:34:19.365 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:34:19.367 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:34:19.405 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:34:20.519 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:36:20.571 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:36:20.613 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:36:21.810 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:36:21.827 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:36:21.827 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:36:21.827 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:36:21.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:36:22.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:36:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:36:22.395 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:36:22.395 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:36:22.424 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:36:22.466 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:36:22.489 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:36:22.490 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:36:22.528 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:36:23.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:38:23.655 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:38:23.693 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:38:25.650 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:38:25.669 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:38:25.669 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:38:25.669 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:38:25.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:38:25.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:38:25.911 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:38:26.153 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:38:26.153 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:38:26.193 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:38:26.222 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:38:26.259 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:38:26.260 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:38:26.308 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:38:27.640 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:40:27.688 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:40:27.727 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:40:28.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:40:28.856 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:40:28.856 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:40:28.857 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:40:28.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:40:29.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:40:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:40:29.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:40:29.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:40:29.382 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:40:29.413 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:40:29.449 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:40:29.450 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:40:29.487 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:40:30.543 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:42:30.592 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:42:30.679 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:42:32.652 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:42:32.690 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:42:32.690 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:42:32.690 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:42:32.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:42:32.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:42:32.966 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:42:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:42:33.316 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:42:33.349 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:42:33.387 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:42:33.416 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:42:33.417 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:42:33.465 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:42:34.573 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:44:34.623 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:44:34.665 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:44:36.644 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:44:36.668 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:44:36.668 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:44:36.669 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:44:36.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:44:36.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:44:36.920 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:44:37.157 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:44:37.157 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:44:37.195 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:44:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:44:37.261 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:44:37.262 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:44:37.305 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:44:38.401 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:46:38.450 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:46:38.490 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:46:39.551 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:46:39.568 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:46:39.568 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:46:39.568 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:46:39.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:46:39.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:46:39.818 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:46:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:46:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:46:40.099 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:46:40.131 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:46:40.165 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:46:40.167 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:46:40.205 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:46:41.267 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:48:41.370 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:48:41.409 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:48:42.470 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:48:42.487 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:48:42.487 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:48:42.487 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:48:42.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:48:42.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:48:42.750 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:48:43.001 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:48:43.001 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:48:43.034 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:48:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:48:43.101 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:48:43.102 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:48:43.145 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:48:44.291 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:50:44.342 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:50:44.380 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:50:46.680 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:50:46.699 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:50:46.699 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:50:46.699 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:50:46.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:50:46.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:50:46.962 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:50:47.192 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:50:47.192 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:50:47.233 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:50:47.262 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:50:47.297 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:50:47.299 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:50:47.345 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:50:48.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:50:48.579 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/ Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 498, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 463, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 320, in report_job response.raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/sage-2021-03-18_16:32:54-rados:cephadm:dashboard-cephadm-iscsi-test-distro-basic-smithi/jobs/5977701/ 2021-03-21T09:52:48.581 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:52:48.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:52:50.691 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:52:50.709 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:52:50.709 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:52:50.709 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:52:50.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:52:50.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:52:50.956 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:52:51.212 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:52:51.212 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:52:51.255 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:52:51.284 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:52:51.323 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:52:51.323 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:52:51.368 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:52:53.060 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:54:53.118 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:54:53.179 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:54:55.313 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:54:55.334 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:54:55.335 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:54:55.335 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:54:55.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:54:55.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:54:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:54:55.845 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:54:55.845 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:54:55.878 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:54:55.917 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:54:55.947 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:54:55.948 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:54:55.996 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:54:58.327 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:56:58.388 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:56:58.427 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:57:00.310 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:57:00.331 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:57:00.331 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:57:00.331 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:57:00.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:57:00.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:57:00.592 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:57:00.821 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:57:00.821 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:57:00.861 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:57:00.889 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:57:00.926 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:57:00.927 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:57:00.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:57:02.062 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:59:02.107 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T09:59:02.688 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:59:06.255 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T09:59:06.274 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T09:59:06.274 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T09:59:06.274 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:59:06.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:59:06.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T09:59:06.524 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T09:59:06.756 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T09:59:06.756 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T09:59:06.791 INFO:teuthology.misc:Compressing logs... 2021-03-21T09:59:06.826 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T09:59:06.858 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T09:59:06.859 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T09:59:06.905 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T09:59:08.013 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:01:08.064 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:01:08.152 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:01:10.398 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:01:10.436 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:01:10.436 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:01:10.436 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:01:10.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:01:10.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:01:10.711 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:01:10.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:01:10.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:01:10.984 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:01:11.024 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:01:11.049 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:01:11.050 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:01:11.090 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:01:13.608 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:03:13.664 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:03:13.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:03:16.242 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:03:16.265 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:03:16.265 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:03:16.265 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:03:16.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:03:16.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:03:16.516 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:03:16.753 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:03:16.753 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:03:16.791 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:03:16.824 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:03:16.855 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:03:16.858 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:03:16.927 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:03:19.000 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:05:19.115 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:05:19.172 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:05:21.284 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:05:21.301 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:05:21.301 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:05:21.301 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:05:21.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:05:21.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:05:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:05:21.775 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:05:21.776 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:05:21.815 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:05:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:05:21.880 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:05:21.881 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:05:21.927 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:05:24.079 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:07:24.136 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:07:24.176 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:07:26.512 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:07:26.532 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:07:26.532 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:07:26.532 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:07:26.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:07:26.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:07:26.798 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:07:27.031 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:07:27.031 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:07:27.059 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:07:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:07:27.125 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:07:27.127 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:07:27.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:07:28.743 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:09:28.801 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:09:28.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:09:30.834 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:09:30.851 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:09:30.852 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:09:30.852 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:09:30.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:09:31.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:09:31.100 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:09:31.336 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:09:31.336 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:09:31.376 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:09:31.405 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:09:31.442 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:09:31.443 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:09:31.483 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:09:32.715 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:11:32.782 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:11:32.822 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:11:34.731 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:11:34.748 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:11:34.748 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:11:34.748 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:11:34.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:11:34.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:11:34.998 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:11:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:11:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:11:35.280 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:11:35.308 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:11:35.347 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:11:35.349 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:11:35.389 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:11:36.919 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:13:36.970 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:13:37.011 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:13:39.003 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:13:39.022 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:13:39.022 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:13:39.022 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:13:39.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:13:39.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:13:39.289 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:13:39.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:13:39.538 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:13:39.570 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:13:39.610 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:13:39.638 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:13:39.638 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:13:39.686 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:13:40.799 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:15:40.864 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:15:40.907 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:15:43.123 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:15:43.139 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:15:43.140 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:15:43.140 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:15:43.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:15:43.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:15:43.389 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:15:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:15:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:15:43.665 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:15:43.698 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:15:43.731 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:15:43.732 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:15:43.850 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:15:45.758 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:17:45.811 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:17:45.864 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:17:47.192 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:17:47.212 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:17:47.212 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:17:47.212 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:17:47.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:17:47.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:17:47.462 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:17:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:17:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:17:47.743 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:17:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:17:47.810 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:17:47.811 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:17:47.856 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:17:48.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:19:48.961 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:19:49.003 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:19:51.245 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:19:51.265 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:19:51.265 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:19:51.266 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:19:51.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:19:51.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:19:51.533 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:19:51.767 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:19:51.767 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:19:51.804 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:19:51.836 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:19:51.869 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:19:51.871 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:19:51.912 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:19:53.166 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:21:53.223 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:21:53.260 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:21:55.066 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:21:55.083 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:21:55.083 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:21:55.083 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:21:55.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:21:55.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:21:55.330 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:21:56.479 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:21:56.480 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:21:56.583 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:21:56.610 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:21:56.612 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:21:56.614 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:21:56.654 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:21:57.721 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:23:57.776 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:23:57.815 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:23:59.890 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:23:59.908 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:23:59.908 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:23:59.908 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:23:59.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:24:00.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:24:00.161 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:24:01.201 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:24:01.201 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:24:01.369 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:24:01.399 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:24:01.403 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:24:01.405 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:24:01.447 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:24:02.547 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:26:02.600 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:26:02.641 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:26:03.739 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:26:03.765 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:26:03.765 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:26:03.765 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:26:03.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:26:03.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:26:04.028 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:26:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:26:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:26:05.162 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:26:05.189 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:26:05.192 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:26:05.195 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:26:05.236 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:26:06.274 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:28:06.328 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:28:06.375 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:28:07.528 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:28:07.549 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:28:07.549 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:28:07.549 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:28:07.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:28:07.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:28:07.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:28:08.558 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:28:08.558 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:28:08.613 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:28:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:28:08.644 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:28:08.646 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:28:08.685 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:28:09.839 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:30:09.899 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:30:09.941 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:30:11.017 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:30:11.040 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:30:11.040 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:30:11.041 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:30:11.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:30:11.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:30:11.291 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:30:11.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:30:11.521 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:30:11.558 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:30:11.591 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:30:11.623 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:30:11.625 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:30:11.665 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:30:12.780 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:32:12.830 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:32:12.874 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:32:14.046 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:32:14.063 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:32:14.064 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:32:14.064 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:32:14.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:32:14.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:32:14.328 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:32:15.226 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:32:15.226 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:32:15.358 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:32:15.387 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:32:15.388 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:32:15.390 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:32:15.428 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:32:16.506 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:34:16.561 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:34:16.600 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:34:17.663 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:34:17.687 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:34:17.687 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:34:17.687 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:34:17.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:34:17.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:34:17.946 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:34:18.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:34:18.833 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:34:18.985 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:34:19.015 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:34:19.018 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:34:19.019 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:34:19.057 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:34:20.103 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:36:20.152 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:36:20.193 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:36:21.225 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:36:21.243 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:36:21.243 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:36:21.243 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:36:21.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:36:21.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:36:21.488 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:36:21.816 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:36:21.816 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:36:21.873 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:36:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:36:21.902 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:36:21.904 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:36:21.943 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:36:22.993 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:38:23.111 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:38:23.153 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:38:24.263 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:38:24.281 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:38:24.281 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:38:24.281 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:38:24.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:38:24.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:38:24.543 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:38:24.786 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:38:24.786 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:38:24.817 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:38:24.854 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:38:24.883 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:38:24.885 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:38:24.922 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:38:25.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:40:26.052 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:40:26.092 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:40:27.132 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:40:27.150 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:40:27.150 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:40:27.150 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:40:27.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:40:27.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:40:27.399 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:40:28.439 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:40:28.440 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:40:28.589 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:40:28.617 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:40:28.618 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:40:28.621 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:40:28.660 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:40:29.799 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:42:29.850 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:42:29.889 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:42:30.972 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:42:30.990 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:42:30.990 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:42:30.990 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:42:30.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:42:31.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:42:31.243 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:42:31.904 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:42:31.904 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:42:31.970 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:42:32.001 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:42:32.006 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:42:32.006 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:42:32.042 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:42:33.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:44:33.249 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:44:33.291 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:44:34.397 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:44:34.414 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:44:34.414 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:44:34.414 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:44:34.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:44:34.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:44:34.677 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:44:34.915 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:44:34.915 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:44:34.943 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:44:34.986 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:44:35.008 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:44:35.010 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:44:35.047 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:44:36.067 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:46:36.117 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:46:36.511 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:46:38.942 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:46:38.961 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:46:38.962 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:46:38.962 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:46:38.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:46:39.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:46:39.281 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:46:40.281 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:46:40.281 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:46:40.355 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:46:40.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:46:40.389 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:46:40.391 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:46:40.435 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:46:41.624 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:48:41.679 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:48:41.718 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:48:42.803 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:48:42.820 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:48:42.820 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:48:42.820 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:48:42.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:48:43.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:48:43.068 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:48:43.737 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:48:43.737 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:48:43.795 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:48:43.825 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:48:43.826 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:48:43.828 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:48:43.871 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:48:44.945 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:50:45.000 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:50:45.090 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:50:46.994 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:50:47.013 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:50:47.013 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:50:47.013 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:50:47.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:50:47.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:50:47.269 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:50:47.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:50:47.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:50:47.553 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:50:47.589 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:50:47.619 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:50:47.621 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:50:47.662 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:50:48.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:52:48.785 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:52:48.823 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:52:49.907 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:52:49.924 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:52:49.924 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:52:49.924 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:52:49.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:52:50.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:52:50.170 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:52:50.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:52:50.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:52:50.914 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:52:50.944 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:52:50.948 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:52:50.950 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:52:50.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:52:52.010 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:54:52.062 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:54:52.100 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:54:53.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:54:53.145 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:54:53.145 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:54:53.145 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:54:53.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:54:53.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:54:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:54:54.153 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:54:54.153 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:54:54.305 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:54:54.332 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:54:54.334 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:54:54.336 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:54:54.374 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:54:55.449 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:56:55.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:56:55.545 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:56:56.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:56:56.657 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:56:56.657 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:56:56.657 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:56:56.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:56:56.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:56:56.917 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:56:57.299 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:56:57.300 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:56:57.503 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:56:57.530 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:56:57.532 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:56:57.535 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:56:57.571 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:56:58.664 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:58:58.718 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T10:58:58.757 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:58:59.827 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T10:58:59.844 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T10:58:59.844 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T10:58:59.844 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:58:59.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:59:00.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T10:59:00.089 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T10:59:00.700 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T10:59:00.700 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T10:59:00.832 INFO:teuthology.misc:Compressing logs... 2021-03-21T10:59:00.863 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T10:59:00.866 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T10:59:00.867 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T10:59:00.914 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T10:59:01.975 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:01:02.030 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:01:02.069 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:01:03.147 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:01:03.165 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:01:03.165 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:01:03.165 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:01:03.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:01:03.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:01:03.417 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:01:03.862 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:01:03.862 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:01:03.987 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:01:04.016 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:01:04.017 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:01:04.020 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:01:04.060 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:01:05.153 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:03:05.206 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:03:05.253 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:03:06.410 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:03:06.428 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:03:06.429 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:03:06.429 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:03:06.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:03:06.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:03:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:03:07.216 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:03:07.216 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:03:07.312 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:03:07.339 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:03:07.341 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:03:07.344 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:03:07.388 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:03:08.488 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:05:08.539 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:05:08.576 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:05:09.760 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:05:09.778 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:05:09.778 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:05:09.778 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:05:09.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:05:09.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:05:10.029 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:05:10.651 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:05:10.651 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:05:10.754 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:05:10.784 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:05:10.785 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:05:10.787 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:05:10.826 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:05:11.909 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:07:11.958 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:07:11.997 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:07:13.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:07:13.222 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:07:13.222 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:07:13.222 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:07:13.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:07:13.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:07:13.474 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:07:14.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:07:14.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:07:14.504 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:07:14.537 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:07:14.543 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:07:14.546 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:07:14.587 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:07:15.707 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:09:15.757 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:09:15.830 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:09:18.166 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:09:18.193 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:09:18.193 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:09:18.193 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:09:18.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:09:18.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:09:18.466 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:09:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:09:19.092 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:09:19.193 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:09:19.220 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:09:19.223 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:09:19.225 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:09:19.265 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:09:20.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:11:20.562 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:11:20.607 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:11:22.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:11:22.608 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:11:22.608 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:11:22.608 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:11:22.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:11:22.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:11:22.859 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:11:24.050 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:11:24.051 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:11:24.295 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:11:24.325 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:11:24.374 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:11:24.376 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:11:24.417 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:11:26.134 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:13:26.197 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:13:26.238 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:13:28.052 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:13:28.071 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:13:28.071 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:13:28.071 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:13:28.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:13:28.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:13:28.320 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:13:28.889 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:13:28.889 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:13:28.955 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:13:28.984 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:13:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:13:28.988 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:13:29.026 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:13:30.636 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:15:30.696 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:15:30.746 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:15:32.635 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:15:32.656 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:15:32.656 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:15:32.657 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:15:32.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:15:32.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:15:32.920 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:15:34.016 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:15:34.016 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:15:34.127 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:15:34.156 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:15:34.158 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:15:34.159 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:15:34.199 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:15:35.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:17:35.336 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:17:35.375 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:17:36.449 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:17:36.466 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:17:36.466 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:17:36.466 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:17:36.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:17:36.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:17:36.720 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:17:37.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:17:37.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:17:37.673 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:17:37.702 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:17:37.706 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:17:37.708 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:17:37.748 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:17:38.888 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:19:38.939 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:19:38.983 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:19:40.021 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:19:40.038 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:19:40.039 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:19:40.039 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:19:40.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:19:40.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:19:40.287 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:19:41.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:19:41.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:19:41.112 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:19:41.142 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:19:41.149 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:19:41.149 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:19:41.188 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:19:42.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:21:42.290 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:21:42.330 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:21:43.440 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:21:43.457 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:21:43.457 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:21:43.457 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:21:43.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:21:43.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:21:43.722 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:21:44.818 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:21:44.818 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:21:44.968 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:21:45.003 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:21:45.007 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:21:45.011 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:21:45.073 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:21:46.263 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:23:46.313 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:23:46.362 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:23:48.444 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:23:48.463 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:23:48.463 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:23:48.464 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:23:48.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:23:48.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:23:48.710 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:23:49.793 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:23:49.793 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:23:49.907 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:23:49.934 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:23:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:23:49.939 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:23:50.032 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:23:51.679 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:25:51.734 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:25:51.825 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:25:53.768 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:25:53.790 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:25:53.790 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:25:53.790 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:25:53.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:25:54.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:25:54.034 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:25:54.265 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:25:54.266 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:25:54.305 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:25:54.334 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:25:54.370 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:25:54.371 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:25:54.411 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:25:56.087 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:27:56.145 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:27:56.191 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:27:58.204 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:27:58.225 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:27:58.226 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:27:58.226 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:27:58.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:27:58.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:27:58.493 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:27:59.599 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:27:59.600 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:27:59.675 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:27:59.703 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:27:59.705 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:27:59.708 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:27:59.745 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:28:00.873 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:30:00.921 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:30:00.960 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:30:02.278 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:30:02.300 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:30:02.300 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:30:02.301 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:30:02.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:30:02.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:30:02.552 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:30:03.738 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:30:03.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:30:03.847 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:30:03.875 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:30:03.877 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:30:03.880 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:30:03.925 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:30:04.995 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:32:05.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:32:05.084 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:32:06.242 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:32:06.260 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:32:06.260 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:32:06.260 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:32:06.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:32:06.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:32:06.504 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:32:06.741 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:32:06.741 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:32:06.781 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:32:06.811 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:32:06.848 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:32:06.849 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:32:06.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:32:07.960 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:34:08.016 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:34:08.063 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:34:10.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:34:10.061 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:34:10.061 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:34:10.061 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:34:10.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:34:10.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:34:10.328 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:34:10.571 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:34:10.571 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:34:10.604 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:34:10.641 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:34:10.671 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:34:10.673 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:34:10.715 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:34:12.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:36:12.326 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:36:12.366 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:36:14.570 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:36:14.586 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:36:14.587 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:36:14.587 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:36:14.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:36:14.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:36:14.832 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:36:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:36:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:36:15.318 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:36:15.348 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:36:15.349 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:36:15.351 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:36:15.403 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:36:16.875 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:38:16.928 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:38:16.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:38:18.975 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:38:18.995 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:38:18.996 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:38:18.996 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:38:18.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:38:19.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:38:19.249 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:38:19.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:38:19.497 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:38:19.541 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:38:19.571 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:38:19.607 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:38:19.608 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:38:19.654 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:38:20.779 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:40:20.828 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:40:20.870 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:40:21.941 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:40:21.960 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:40:21.960 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:40:21.960 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:40:21.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:40:22.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:40:22.220 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:40:22.467 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:40:22.467 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:40:22.506 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:40:22.535 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:40:22.573 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:40:22.573 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:40:22.617 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:40:23.675 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:42:23.726 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:42:23.769 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:42:25.584 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:42:25.601 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:42:25.602 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:42:25.602 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:42:25.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:42:25.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:42:25.849 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:42:26.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:42:26.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:42:26.129 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:42:26.161 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:42:26.195 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:42:26.196 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:42:26.238 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:42:27.368 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:44:27.421 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:44:27.461 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:44:29.314 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:44:29.333 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:44:29.333 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:44:29.334 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:44:29.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:44:29.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:44:29.578 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:44:29.821 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:44:29.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:44:29.860 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:44:29.892 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:44:29.924 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:44:29.925 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:44:29.963 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:44:31.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:46:31.176 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:46:31.215 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:46:33.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:46:33.235 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:46:33.235 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:46:33.235 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:46:33.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:46:33.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:46:33.494 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:46:33.732 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:46:33.733 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:46:33.760 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:46:33.800 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:46:33.825 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:46:33.827 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:46:33.885 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:46:35.004 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:48:35.055 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:48:35.093 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:48:36.989 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:48:37.008 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:48:37.008 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:48:37.008 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:48:37.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:48:37.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:48:37.254 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:48:37.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:48:37.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:48:37.529 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:48:37.562 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:48:37.595 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:48:37.597 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:48:37.636 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:48:38.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:50:38.897 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:50:38.937 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:50:40.741 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:50:40.759 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:50:40.760 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:50:40.760 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:50:40.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:50:40.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:50:41.011 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:50:41.763 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:50:41.763 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:50:41.946 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:50:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:50:41.979 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:50:41.980 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:50:42.019 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:50:43.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:52:43.118 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:52:43.156 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:52:44.238 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:52:44.259 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:52:44.260 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:52:44.260 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:52:44.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:52:44.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:52:44.529 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:52:45.260 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:52:45.260 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:52:45.371 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:52:45.400 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:52:45.402 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:52:45.404 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:52:45.447 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:52:46.537 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:54:46.598 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:54:46.635 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:54:47.838 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:54:47.856 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:54:47.856 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:54:47.856 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:54:47.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:54:48.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:54:48.106 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:54:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:54:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:54:49.098 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:54:49.127 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:54:49.128 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:54:49.131 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:54:49.170 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:54:50.248 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:56:50.312 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:56:50.352 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:56:51.444 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:56:51.462 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:56:51.462 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:56:51.462 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:56:51.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:56:51.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:56:51.710 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:56:52.677 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:56:52.677 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:56:52.763 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:56:52.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:56:52.794 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:56:52.796 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:56:52.833 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:56:53.937 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:58:53.987 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:58:54.030 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:58:55.069 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T11:58:55.088 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T11:58:55.088 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T11:58:55.088 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:58:55.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:58:55.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:58:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T11:58:56.236 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T11:58:56.236 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T11:58:56.351 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:58:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T11:58:56.381 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T11:58:56.383 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T11:58:56.474 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T11:58:57.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:00:57.589 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:00:57.637 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:00:58.751 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:00:58.768 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:00:58.768 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:00:58.768 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:00:58.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:00:58.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:00:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:00:59.931 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:00:59.931 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:01:00.079 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:01:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:01:00.113 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:01:00.115 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:01:00.159 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:01:01.308 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:03:01.362 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:03:01.402 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:03:02.465 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:03:02.482 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:03:02.482 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:03:02.482 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:03:02.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:03:02.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:03:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:03:03.097 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:03:03.097 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:03:03.145 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:03:03.173 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:03:03.175 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:03:03.178 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:03:03.216 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:03:04.276 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:05:04.325 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:05:04.364 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:05:05.425 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:05:05.442 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:05:05.442 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:05:05.442 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:05:05.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:05:05.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:05:05.705 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:05:06.193 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:05:06.193 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:05:06.300 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:05:06.329 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:05:06.330 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:05:06.333 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:05:06.371 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:05:07.455 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:07:07.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:07:07.544 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:07:08.768 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:07:08.787 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:07:08.787 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:07:08.787 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:07:08.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:07:09.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:07:09.033 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:07:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:07:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:07:09.791 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:07:09.821 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:07:09.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:07:09.824 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:07:09.870 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:07:10.955 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:09:11.128 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:09:11.177 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:09:12.312 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:09:12.328 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:09:12.329 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:09:12.329 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:09:12.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:09:12.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:09:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:09:13.280 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:09:13.281 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:09:13.387 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:09:13.417 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:09:13.418 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:09:13.420 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:09:13.458 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:09:14.513 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:11:14.565 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:11:14.604 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:11:15.705 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:11:15.727 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:11:15.727 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:11:15.727 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:11:15.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:11:15.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:11:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:11:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:11:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:11:16.776 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:11:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:11:16.808 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:11:16.810 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:11:16.848 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:11:17.911 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:13:17.963 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:13:18.007 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:13:19.135 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:13:19.152 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:13:19.153 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:13:19.153 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:13:19.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:13:19.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:13:19.403 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:13:20.101 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:13:20.101 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:13:20.216 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:13:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:13:20.246 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:13:20.248 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:13:20.292 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:13:21.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:15:21.483 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:15:21.522 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:15:22.596 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:15:22.620 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:15:22.621 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:15:22.621 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:15:22.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:15:22.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:15:22.864 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:15:23.554 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:15:23.554 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:15:23.631 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:15:23.661 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:15:23.666 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:15:23.668 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:15:23.706 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:15:24.841 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:17:24.894 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:17:24.948 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:17:26.134 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:17:26.151 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:17:26.152 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:17:26.152 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:17:26.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:17:26.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:17:26.417 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:17:26.769 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:17:26.769 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:17:26.820 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:17:26.848 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:17:26.849 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:17:26.852 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:17:26.894 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:17:27.956 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:19:28.012 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:19:28.050 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:19:29.176 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:19:29.193 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:19:29.193 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:19:29.193 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:19:29.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:19:29.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:19:29.440 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:19:30.064 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:19:30.064 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:19:30.125 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:19:30.154 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:19:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:19:30.158 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:19:30.195 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:19:31.258 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:21:31.309 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:21:31.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:21:32.443 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:21:32.460 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:21:32.460 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:21:32.460 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:21:32.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:21:32.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:21:32.704 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:21:33.320 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:21:33.321 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:21:33.481 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:21:33.512 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:21:33.513 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:21:33.515 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:21:33.552 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:21:34.687 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:23:34.739 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:23:34.779 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:23:35.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:23:35.907 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:23:35.907 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:23:35.907 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:23:35.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:23:36.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:23:36.174 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:23:36.767 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:23:36.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:23:36.903 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:23:36.936 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:23:36.937 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:23:36.938 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:23:36.990 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:23:38.036 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:25:38.089 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:25:38.143 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:25:39.264 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:25:39.282 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:25:39.282 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:25:39.282 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:25:39.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:25:39.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:25:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:25:39.910 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:25:39.911 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:25:39.958 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:25:39.988 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:25:40.024 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:25:40.026 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:25:40.064 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:25:41.084 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:27:41.134 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:27:41.178 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:27:42.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:27:42.292 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:27:42.292 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:27:42.292 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:27:42.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:27:42.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:27:42.543 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:27:43.164 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:27:43.164 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:27:43.270 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:27:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:27:43.300 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:27:43.303 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:27:43.340 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:27:44.387 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:29:44.444 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:29:44.505 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:29:45.663 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:29:45.680 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:29:45.680 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:29:45.680 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:29:45.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:29:45.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:29:45.946 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:29:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:29:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:29:46.512 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:29:46.541 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:29:46.542 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:29:46.544 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:29:46.588 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:29:47.696 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:31:47.749 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:31:47.789 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:31:48.825 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:31:48.842 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:31:48.842 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:31:48.842 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:31:48.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:31:49.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:31:49.089 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:31:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:31:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:31:49.885 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:31:49.913 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:31:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:31:49.917 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:31:49.956 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:31:50.998 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:33:51.056 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:33:51.154 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:33:52.301 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:33:52.318 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:33:52.318 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:33:52.318 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:33:52.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:33:52.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:33:52.564 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:33:54.115 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:33:54.115 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:33:54.303 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:33:54.334 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:33:54.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:33:54.337 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:33:54.374 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:33:55.515 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:35:55.582 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:35:55.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:35:56.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:35:56.669 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:35:56.670 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:35:56.670 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:35:56.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:35:56.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:35:56.931 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:35:57.660 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:35:57.660 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:35:57.729 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:35:57.759 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:35:57.760 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:35:57.763 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:35:57.805 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:35:58.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:37:58.844 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:37:58.882 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:37:59.883 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:37:59.900 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:37:59.901 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:37:59.901 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:37:59.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:38:00.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:38:00.145 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:38:01.383 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:38:01.383 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:38:01.608 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:38:01.638 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:38:01.641 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:38:01.643 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:38:01.680 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:38:02.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:40:02.754 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:40:02.795 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:40:03.819 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:40:03.836 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:40:03.836 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:40:03.836 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:40:03.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:40:04.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:40:04.082 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:40:04.455 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:40:04.455 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:40:04.517 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:40:04.545 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:40:04.546 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:40:04.550 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:40:04.587 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:40:05.586 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:42:05.653 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:42:05.718 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:42:06.784 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:42:06.801 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:42:06.801 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:42:06.801 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:42:06.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:42:07.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:42:07.065 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:42:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:42:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:42:07.722 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:42:07.751 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:42:07.753 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:42:07.755 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:42:07.793 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:42:08.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:44:08.871 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:44:08.909 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:44:09.941 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:44:09.958 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:44:09.958 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:44:09.959 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:44:09.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:44:10.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:44:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:44:11.475 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:44:11.475 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:44:11.716 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:44:11.746 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:44:11.804 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:44:11.806 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:44:11.859 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:44:12.986 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:46:13.037 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:46:13.077 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:46:14.199 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:46:14.216 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:46:14.216 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:46:14.216 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:46:14.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:46:14.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:46:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:46:15.313 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:46:15.314 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:46:15.441 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:46:15.472 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:46:15.501 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:46:15.503 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:46:15.541 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:46:16.619 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:48:16.671 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:48:16.713 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:48:18.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:48:18.855 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:48:18.855 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:48:18.855 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:48:18.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:48:19.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:48:19.115 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:48:19.860 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:48:19.860 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:48:19.952 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:48:19.981 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:48:20.005 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:48:20.007 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:48:20.076 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:48:21.466 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:50:21.543 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:50:21.586 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:50:23.524 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:50:23.541 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:50:23.541 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:50:23.542 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:50:23.542 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:50:23.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:50:23.790 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:50:24.866 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:50:24.866 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:50:25.013 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:50:25.042 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:50:25.111 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:50:25.112 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:50:25.172 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:50:26.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:52:26.311 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:52:26.352 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:52:27.399 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:52:27.416 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:52:27.417 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:52:27.417 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:52:27.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:52:27.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:52:27.662 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:52:30.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:52:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:52:30.783 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:52:30.812 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:52:30.970 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:52:30.972 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:52:31.009 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:52:32.090 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:54:32.138 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:54:32.177 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:54:33.342 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:54:33.360 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:54:33.360 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:54:33.361 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:54:33.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:54:33.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:54:33.628 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:54:35.027 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:54:35.027 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:54:35.230 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:54:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:54:35.298 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:54:35.300 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:54:35.337 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:54:36.341 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:56:36.393 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:56:36.432 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:56:37.486 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:56:37.503 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:56:37.503 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:56:37.503 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:56:37.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:56:37.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:56:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:56:37.995 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:56:37.995 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:56:38.034 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:56:38.064 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:56:38.100 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T12:56:38.101 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T12:56:38.138 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:56:39.270 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:58:39.325 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T12:58:39.363 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:58:40.371 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T12:58:40.412 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T12:58:40.412 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T12:58:40.413 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:58:40.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:58:40.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T12:58:40.671 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T12:58:40.904 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T12:58:40.904 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:58:40.944 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:58:40.973 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:58:41.008 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T12:58:41.041 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T12:58:41.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:58:41.074 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T12:58:41.075 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:58:41.076 INFO:teuthology.misc:Compressing logs... 2021-03-21T12:58:41.110 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T12:58:41.178 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T12:58:41.178 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T12:58:41.245 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T12:58:42.248 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:00:42.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:00:42.338 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:00:43.375 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:00:43.392 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:00:43.392 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:00:43.392 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:00:43.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:00:43.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:00:43.655 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:01:02.018 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T13:01:02.058 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:01:03.067 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:03:03.118 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:03:03.158 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:03:04.253 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:03:04.270 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:03:04.270 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:03:04.270 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:03:04.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:03:04.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:03:04.522 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:03:07.555 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T13:03:07.599 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:03:08.704 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:05:08.758 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:05:08.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:05:09.944 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:05:09.972 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:05:09.972 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:05:09.972 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:05:09.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:05:10.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:05:10.221 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:05:10.251 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T13:05:11.308 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T13:05:12.367 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T13:05:13.463 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T13:05:14.563 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T13:05:15.622 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T13:05:16.755 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T13:05:17.814 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T13:05:18.913 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T13:05:20.014 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T13:05:20.014 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T13:05:20.052 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:05:21.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:07:21.194 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:07:21.235 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:07:22.291 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:07:22.308 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:07:22.309 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:07:22.309 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:07:22.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:07:22.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:07:22.572 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:08:22.530 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T13:08:22.623 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:08:23.733 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:10:23.784 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:10:23.823 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:10:25.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:10:25.852 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:10:25.852 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:10:25.852 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:10:25.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:10:26.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:10:26.102 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:10:57.938 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T13:10:57.971 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:10:57.971 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:10:57.997 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:10:57.997 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:10:58.000 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:10:58.042 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:10:58.058 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T13:10:58.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T13:10:58.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:10:58.130 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T13:10:58.130 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:10:58.133 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:10:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T13:10:58.190 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T13:10:58.249 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:10:58.249 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:10:58.266 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:10:58.266 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:10:58.305 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:11:00.114 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:13:00.166 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:13:01.837 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:13:03.712 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:13:03.731 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:13:03.731 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:13:03.731 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:13:03.732 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:13:03.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:13:03.991 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:13:04.257 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T13:13:04.289 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:13:04.289 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:13:04.317 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:13:04.317 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:13:04.319 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:13:04.357 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:13:04.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T13:13:04.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:13:04.462 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:13:04.492 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T13:13:04.505 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T13:13:04.560 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:13:04.560 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:13:04.575 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:13:04.576 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:13:04.615 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:13:06.076 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:15:06.138 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:15:06.177 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:15:08.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:15:08.090 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:15:08.090 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:15:08.090 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:15:08.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:15:08.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:15:08.339 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:15:08.657 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:15:08.657 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:15:08.686 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:15:08.728 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:15:08.797 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T13:15:08.797 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:15:08.838 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:15:08.867 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T13:15:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:15:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:15:09.060 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:15:10.147 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:17:10.198 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:17:10.235 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:17:11.853 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:17:11.870 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:17:11.871 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:17:11.871 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:17:11.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:17:12.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:17:12.119 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:17:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:17:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:17:12.475 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:17:12.515 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:17:12.585 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T13:17:12.585 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:17:12.625 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:17:12.655 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T13:17:12.775 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2021-03-21T13:17:12.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:17:12.829 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:17:12.972 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:17:14.042 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:19:14.090 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:19:14.151 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:19:15.278 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:19:15.295 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:19:15.295 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:19:15.295 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:19:15.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:19:15.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:19:15.560 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:19:34.083 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T13:19:34.123 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:19:35.271 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:21:35.322 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:21:35.360 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:21:36.423 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:21:36.441 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:21:36.441 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:21:36.441 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:21:36.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:21:36.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:21:36.689 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:21:37.830 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T13:21:37.869 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:21:38.883 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:23:38.938 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:23:38.976 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:23:40.037 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:23:40.054 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:23:40.054 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:23:40.054 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:23:40.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:23:40.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:23:40.301 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:23:40.516 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:23:40.546 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:23:40.546 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:23:40.592 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:23:40.592 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:23:40.594 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:23:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:23:40.658 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:23:40.690 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T13:23:40.690 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:23:40.725 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T13:23:40.725 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:23:40.726 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:23:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T13:23:40.791 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T13:23:40.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:23:40.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:23:40.856 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:23:40.856 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:23:40.891 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:23:41.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:25:41.939 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:25:41.978 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:25:43.785 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:25:43.804 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:25:43.805 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:25:43.805 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:25:43.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:25:44.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:25:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:25:44.263 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:25:44.294 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:25:44.294 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:25:44.329 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:25:44.329 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:25:44.330 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:25:44.364 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:25:44.434 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T13:25:44.434 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:25:44.477 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:25:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T13:25:44.542 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T13:25:44.574 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:25:44.574 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:25:44.608 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:25:44.608 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:25:44.646 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:25:45.732 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:27:45.784 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:27:45.824 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:27:46.953 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:27:46.970 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:27:46.970 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:27:46.970 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:27:46.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:27:47.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:27:47.223 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:27:47.424 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:27:47.453 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:27:47.453 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:27:47.492 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:27:47.492 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:27:47.493 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:27:47.521 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:27:47.593 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T13:27:47.593 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:27:47.648 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:27:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T13:27:47.690 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T13:27:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:27:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:27:47.766 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T13:27:47.766 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:27:47.809 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:27:48.891 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:29:48.941 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:29:48.979 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:29:50.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:29:50.956 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:29:50.956 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:29:50.956 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:29:50.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:29:51.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:29:51.203 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:29:51.405 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:29:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:29:51.435 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:29:51.470 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:29:51.470 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:29:51.473 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:29:51.505 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:29:51.537 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:29:51.539 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:29:51.583 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:29:52.662 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:31:52.721 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:31:52.792 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:31:54.795 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:31:54.814 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:31:54.814 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:31:54.814 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:31:54.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:31:55.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:31:55.080 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:31:55.318 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:31:55.319 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:31:55.355 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:31:55.389 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:31:55.420 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:31:55.421 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:31:55.470 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:31:57.390 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:33:57.446 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:33:57.510 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:33:59.393 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:33:59.410 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:33:59.410 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:33:59.410 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:33:59.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:33:59.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:33:59.662 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:33:59.892 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:33:59.893 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:33:59.930 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:33:59.964 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:33:59.998 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:34:00.000 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:34:00.041 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:34:01.773 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:36:01.825 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:36:01.888 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:36:03.882 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:36:03.899 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:36:03.899 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:36:03.899 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:36:03.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:36:04.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:36:04.148 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:36:04.380 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:36:04.381 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:36:04.418 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:36:04.451 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:36:04.486 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:36:04.488 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:36:04.559 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:36:06.517 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:38:06.567 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:38:06.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:38:08.378 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:38:08.397 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:38:08.397 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:38:08.398 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:38:08.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:38:08.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:38:08.662 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:38:08.902 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:38:08.902 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:38:08.933 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:38:08.972 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:38:08.999 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:38:09.000 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:38:09.039 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:38:10.079 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:40:10.130 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:40:10.202 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:40:12.132 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:40:12.149 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:40:12.149 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:40:12.150 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:40:12.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:40:12.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:40:12.394 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:40:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:40:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:40:12.671 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:40:12.698 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:40:12.736 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:40:12.737 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:40:12.777 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:40:14.437 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:42:14.505 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:42:14.559 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:42:16.518 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:42:16.535 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:42:16.535 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:42:16.536 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:42:16.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:42:16.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:42:16.784 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:42:17.024 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:42:17.024 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:42:17.066 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:42:17.093 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:42:17.133 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:42:17.134 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:42:17.173 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:42:18.288 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:44:18.343 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:44:18.382 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:44:20.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:44:20.226 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:44:20.226 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:44:20.226 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:44:20.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:44:20.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:44:20.497 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:44:20.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:44:20.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:44:20.760 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:44:20.799 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:44:20.827 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:44:20.829 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:44:20.909 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:44:22.998 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:46:23.129 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:46:23.170 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:46:25.141 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:46:25.158 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:46:25.158 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:46:25.159 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:46:25.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:46:25.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:46:25.404 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:46:25.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:46:25.637 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:46:25.675 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:46:25.706 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:46:25.739 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:46:25.740 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:46:25.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:46:26.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:48:26.918 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:48:26.986 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:48:29.015 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:48:29.032 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:48:29.033 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:48:29.033 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:48:29.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:48:29.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:48:29.287 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:48:29.525 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:48:29.525 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:48:29.565 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:48:29.595 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:48:29.630 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:48:29.632 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:48:29.684 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:48:31.508 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:50:31.575 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:50:31.659 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:50:33.565 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:50:33.582 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:50:33.582 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:50:33.582 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:50:33.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:50:33.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:50:33.845 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:50:34.081 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:50:34.082 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:50:34.110 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:50:34.150 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:50:34.175 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:50:34.176 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:50:34.243 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:50:35.393 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:52:35.444 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:52:35.478 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:52:37.329 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:52:37.345 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:52:37.346 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:52:37.346 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:52:37.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:52:37.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:52:37.599 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:52:37.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:52:37.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:52:37.879 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:52:37.908 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:52:37.945 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:52:37.946 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:52:37.985 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:52:38.993 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:54:39.042 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:54:39.080 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:54:40.125 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:54:40.142 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:54:40.143 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:54:40.143 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:54:40.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:54:40.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:54:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:54:40.633 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:54:40.633 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:54:40.669 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:54:40.704 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:54:40.734 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:54:40.736 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:54:40.773 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:54:41.801 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:56:41.852 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:56:41.888 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:56:42.875 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:56:42.892 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:56:42.892 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:56:42.892 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:56:42.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:56:43.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:56:43.154 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:56:43.373 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:56:43.374 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:56:43.414 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:56:43.441 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:56:43.480 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:56:43.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:56:43.519 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:56:44.460 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:58:44.511 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T13:58:44.552 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:58:45.606 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T13:58:45.623 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T13:58:45.623 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T13:58:45.623 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:58:45.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:58:45.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T13:58:45.868 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T13:58:46.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T13:58:46.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T13:58:46.129 INFO:teuthology.misc:Compressing logs... 2021-03-21T13:58:46.159 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T13:58:46.194 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T13:58:46.195 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T13:58:46.229 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T13:58:47.211 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:00:47.259 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:00:47.296 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:00:48.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:00:48.367 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:00:48.367 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:00:48.367 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:00:48.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:00:48.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:00:48.616 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:01:07.011 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T14:01:07.047 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:01:07.999 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:03:08.051 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:03:08.091 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:03:09.160 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:03:09.178 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:03:09.178 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:03:09.178 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:03:09.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:03:09.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:03:09.445 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:03:09.459 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T14:03:10.556 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T14:03:11.655 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T14:03:12.753 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T14:03:13.810 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T14:03:14.908 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T14:03:16.006 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T14:03:17.105 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T14:03:18.200 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T14:03:19.298 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T14:03:19.299 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T14:03:19.337 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:03:20.390 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:05:20.448 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:05:20.493 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:05:21.577 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:05:21.594 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:05:21.594 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:05:21.594 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:05:21.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:05:21.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:05:21.850 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:05:22.690 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T14:05:22.728 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:05:23.748 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:07:23.801 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:07:23.836 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:07:24.770 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:07:24.786 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:07:24.786 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:07:24.787 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:07:24.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:07:25.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:07:25.035 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:07:25.328 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T14:07:25.359 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:07:25.359 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:07:25.387 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:07:25.388 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:07:25.390 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:07:25.432 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:07:25.448 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T14:07:25.502 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:07:25.503 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:07:25.521 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:07:25.522 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:07:25.524 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:07:25.572 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:07:25.580 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T14:07:25.641 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:07:25.641 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:07:25.657 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:07:25.657 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:07:25.696 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:07:26.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:09:26.795 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:09:26.835 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:09:27.840 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:09:27.856 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:09:27.856 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:09:27.856 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:09:27.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:09:28.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:09:28.126 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:09:28.433 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T14:09:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:09:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:09:28.489 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:09:28.489 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:09:28.491 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:09:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:09:28.547 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T14:09:28.606 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:09:28.607 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:09:28.622 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:09:28.622 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:09:28.624 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:09:28.677 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:09:28.679 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T14:09:28.745 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:09:28.746 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:09:28.761 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:09:28.761 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:09:28.800 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:09:29.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:11:29.877 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:11:29.911 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:11:30.904 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:11:30.920 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:11:30.920 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:11:30.920 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:11:30.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:11:31.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:11:31.166 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:11:31.469 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T14:11:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:11:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:11:31.530 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:11:31.530 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:11:31.531 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:11:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:11:31.640 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:11:31.640 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:11:31.669 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:11:31.707 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:11:31.713 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T14:11:31.774 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:11:31.775 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:11:31.790 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:11:31.790 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:11:31.829 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:11:32.822 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:13:32.874 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:13:32.909 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:13:33.870 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:13:33.886 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:13:33.886 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:13:33.887 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:13:33.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:13:34.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:13:34.137 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:13:34.423 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T14:13:34.454 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:13:34.454 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:13:34.481 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:13:34.482 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:13:34.483 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:13:34.526 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:13:34.597 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:13:34.597 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:13:34.630 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:13:34.666 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:13:34.669 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T14:13:34.735 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:13:34.735 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:13:34.750 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:13:34.750 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:13:34.788 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:13:35.768 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:15:35.822 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:15:35.860 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:15:36.933 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:15:36.949 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:15:36.949 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:15:36.949 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:15:36.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:15:37.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:15:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:15:37.539 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:15:37.539 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:15:37.569 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:15:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:15:37.681 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:15:37.681 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:15:37.721 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:15:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:15:37.819 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:15:37.819 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:15:37.873 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:15:38.891 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:17:38.943 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:17:38.984 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:17:40.055 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:17:40.076 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:17:40.077 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:17:40.077 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:17:40.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:17:40.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:17:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:17:40.652 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:17:40.652 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:17:40.681 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:17:40.727 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:17:40.799 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:17:40.799 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:17:40.835 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:17:40.867 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:17:40.936 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:17:40.936 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:17:41.048 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:17:42.152 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:19:42.205 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:19:42.243 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:19:43.242 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:19:43.258 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:19:43.258 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:19:43.258 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:19:43.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:19:43.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:19:43.509 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:19:43.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:19:43.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:19:43.880 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:19:43.922 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:19:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:19:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:19:44.032 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:19:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:19:44.132 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:19:44.132 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:19:44.193 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:19:45.186 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:21:45.238 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:21:45.276 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:21:46.394 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:21:46.413 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:21:46.413 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:21:46.413 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:21:46.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:21:46.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:21:46.678 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:21:47.004 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:21:47.005 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:21:47.036 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:21:47.073 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:21:47.142 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:21:47.142 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:21:47.181 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:21:47.208 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:21:47.276 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:21:47.276 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:21:47.342 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:21:48.431 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:23:48.486 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:23:48.525 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:23:49.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:23:49.557 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:23:49.557 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:23:49.558 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:23:49.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:23:49.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:23:49.805 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:23:50.145 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:23:50.145 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:23:50.176 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:23:50.214 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:23:50.282 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:23:50.282 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:23:50.318 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:23:50.350 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:23:50.418 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:23:50.418 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:23:50.483 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:23:51.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:25:51.552 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:25:51.590 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:25:52.561 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:25:52.578 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:25:52.578 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:25:52.578 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:25:52.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:25:52.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:25:52.830 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:25:53.167 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:25:53.167 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:25:53.198 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:25:53.238 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:25:53.308 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:25:53.308 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:25:53.343 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:25:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:25:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:25:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:25:53.513 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:25:54.568 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:27:54.617 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:27:54.669 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:27:55.763 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:27:55.782 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:27:55.782 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:27:55.782 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:27:55.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:27:56.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:27:56.045 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:27:56.374 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:27:56.375 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:27:56.407 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:27:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:27:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:27:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:27:56.555 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:27:56.582 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:27:56.651 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:27:56.651 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:27:56.716 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:27:57.754 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:29:57.805 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:29:57.843 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:29:58.843 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:29:58.860 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:29:58.860 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:29:58.860 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:29:58.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:29:59.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:29:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:30:17.538 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T14:30:17.573 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:30:18.571 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:32:18.622 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:32:18.662 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:32:19.763 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:32:19.780 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:32:19.781 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:32:19.781 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:32:19.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:32:20.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:32:20.031 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:32:38.338 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T14:32:38.375 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:32:39.413 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:34:39.465 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:34:39.504 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:34:40.914 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:34:40.931 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:34:40.931 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:34:40.931 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:34:40.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:34:41.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:34:41.192 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:34:42.118 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T14:34:42.159 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:34:43.277 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:36:43.326 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:36:43.365 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:36:44.404 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:36:44.422 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:36:44.422 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:36:44.422 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:36:44.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:36:44.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:36:44.669 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:36:44.967 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:36:45.007 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:36:45.007 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:36:45.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:36:45.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:36:45.070 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:36:45.100 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:36:45.164 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:36:45.194 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:36:45.194 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:36:45.247 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:36:45.247 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:36:45.248 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:36:45.279 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:36:45.320 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:36:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:36:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:36:45.389 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:36:45.389 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:36:45.461 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:36:46.485 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:38:46.534 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:38:46.571 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:38:47.634 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:38:47.655 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:38:47.656 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:38:47.657 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:38:47.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:38:47.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:38:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:38:48.104 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:38:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:38:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:38:48.171 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:38:48.171 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:38:48.172 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:38:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:38:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T14:38:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:38:48.332 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:38:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:38:48.375 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T14:38:48.427 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:38:48.428 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:38:48.451 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T14:38:48.452 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:38:48.510 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:38:49.621 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:40:49.682 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:40:49.716 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:40:50.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:40:50.747 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:40:50.748 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:40:50.748 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:40:50.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:40:50.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:40:51.010 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:40:51.200 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:40:51.231 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:40:51.231 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:40:51.266 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:40:51.266 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:40:51.267 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:40:51.301 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:40:51.334 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T14:40:51.336 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T14:40:51.375 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:40:52.411 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:42:52.500 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:42:52.536 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:42:53.510 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:42:53.527 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:42:53.527 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:42:53.528 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:42:53.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:42:53.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:42:53.773 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:42:53.970 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:42:53.999 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:42:53.999 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:42:54.035 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:42:54.035 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:42:54.037 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:42:54.071 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:42:54.102 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T14:42:54.103 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T14:42:54.141 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:42:55.152 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:44:55.204 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:44:55.238 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:44:56.224 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:44:56.241 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:44:56.241 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:44:56.241 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:44:56.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:44:56.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:44:56.486 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:44:56.721 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:44:56.721 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:44:56.759 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:44:56.791 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:44:56.824 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T14:44:56.825 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T14:44:56.859 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:44:57.800 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:46:57.855 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:46:57.893 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:46:58.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:46:58.951 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:46:58.951 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:46:58.951 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:46:58.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:46:59.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:46:59.213 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:46:59.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:46:59.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:46:59.465 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:46:59.494 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:46:59.528 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T14:46:59.530 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T14:46:59.569 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:47:00.587 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:49:00.637 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:49:00.674 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:49:01.656 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:49:01.673 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:49:01.673 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:49:01.673 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:49:01.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:49:01.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:49:01.924 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:49:02.165 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:49:02.166 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:49:02.203 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:49:02.235 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:49:02.268 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T14:49:02.270 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T14:49:02.306 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:49:03.326 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:51:03.429 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:51:03.466 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:51:04.440 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:51:04.457 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:51:04.457 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:51:04.458 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:51:04.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:51:04.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:51:04.705 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:51:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:51:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:51:04.978 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:51:05.021 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:51:05.043 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T14:51:05.044 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T14:51:05.081 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:51:06.080 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:53:06.131 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:53:06.166 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:53:07.109 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:53:07.125 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:53:07.125 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:53:07.126 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:53:07.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:53:07.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:53:07.387 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:53:07.613 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:53:07.614 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:53:07.655 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:53:07.681 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:53:07.721 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T14:53:07.723 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T14:53:07.795 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:53:08.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:55:08.827 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:55:08.864 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:55:09.961 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:55:09.979 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:55:09.980 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:55:09.980 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:55:09.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:55:10.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:55:10.232 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:55:10.471 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T14:55:10.471 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T14:55:10.509 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:55:10.541 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T14:55:10.574 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T14:55:10.576 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T14:55:10.612 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:55:11.787 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:57:11.846 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:57:11.880 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:57:12.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:57:12.887 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:57:12.887 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:57:12.888 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:57:12.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:57:13.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:57:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:57:31.586 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T14:57:31.625 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:57:32.616 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:59:32.686 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T14:59:32.722 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T14:59:33.729 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T14:59:33.768 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T14:59:33.768 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T14:59:33.768 INFO:teuthology.misc:Compressing logs... 2021-03-21T14:59:33.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:59:33.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T14:59:34.031 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T14:59:34.046 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T14:59:35.144 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:00:36.147 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T15:00:36.185 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:00:37.194 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:02:37.282 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:02:37.319 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:02:38.329 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:02:38.344 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:02:38.344 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:02:38.345 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:02:38.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:02:38.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:02:38.597 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:02:38.863 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:02:38.894 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:02:38.894 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:02:38.963 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:02:38.963 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:02:38.965 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:02:38.992 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:02:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:02:39.064 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:02:39.064 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:02:39.089 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:02:39.089 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:02:39.090 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:02:39.132 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:02:39.153 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:02:39.201 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:02:39.201 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:02:39.225 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:02:39.225 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:02:39.258 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:02:40.304 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:04:40.360 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:04:40.396 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:04:41.368 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:04:41.384 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:04:41.384 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:04:41.384 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:04:41.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:04:41.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:04:41.644 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:04:41.816 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:04:41.845 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:04:41.845 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:04:41.880 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:04:41.880 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:04:41.881 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:04:41.913 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:04:41.946 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:04:41.983 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:04:41.983 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:04:42.012 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:04:42.012 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:04:42.014 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:04:42.050 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:04:42.079 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:04:42.119 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:04:42.119 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:04:42.146 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:04:42.146 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:04:42.181 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:04:43.186 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:06:43.237 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:06:43.273 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:06:44.323 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:06:44.339 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:06:44.339 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:06:44.339 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:06:44.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:06:44.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:06:44.585 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:06:45.366 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:06:45.398 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:06:45.398 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:06:45.432 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:06:45.432 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:06:45.434 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:06:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:06:45.499 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:06:45.539 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:06:45.539 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:06:45.568 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:06:45.569 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:06:45.570 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:06:45.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:06:45.639 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:06:45.677 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:06:45.677 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:06:45.705 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:06:45.706 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:06:45.743 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:06:46.800 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:08:46.862 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:08:46.900 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:08:47.950 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:08:47.966 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:08:47.966 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:08:47.967 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:08:47.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:08:48.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:08:48.220 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:08:48.427 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:08:48.458 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:08:48.458 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:08:48.493 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:08:48.493 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:08:48.494 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:08:48.529 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:08:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:08:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:08:48.647 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:08:48.673 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:08:48.701 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:08:48.743 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:08:48.743 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:08:48.769 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:08:48.769 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:08:48.808 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:08:49.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:10:49.870 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:10:49.905 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:10:50.958 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:10:50.975 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:10:50.975 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:10:50.975 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:10:50.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:10:51.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:10:51.235 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:10:51.417 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:10:51.446 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:10:51.446 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:10:51.483 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:10:51.483 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:10:51.485 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:10:51.516 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:10:51.551 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T15:10:51.552 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T15:10:51.589 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:10:52.617 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:12:52.668 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:12:52.704 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:12:53.699 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:12:53.715 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:12:53.716 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:12:53.716 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:12:53.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:12:53.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:12:53.962 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:12:54.169 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:12:54.199 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:12:54.199 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:12:54.237 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:12:54.237 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:12:54.239 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:12:54.268 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:12:54.305 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T15:12:54.306 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T15:12:54.341 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:12:55.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:14:55.311 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:14:55.349 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:14:56.457 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:14:56.476 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:14:56.476 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:14:56.476 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:14:56.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:14:56.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:14:56.724 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:14:56.929 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:14:56.960 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:14:56.960 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:14:56.997 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:14:56.997 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:14:56.998 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:14:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:14:57.063 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T15:14:57.065 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T15:14:57.103 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:14:58.214 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:16:58.267 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:16:58.303 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:16:59.271 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:16:59.286 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:16:59.287 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:16:59.287 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:16:59.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:16:59.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:16:59.547 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:16:59.735 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:16:59.765 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:16:59.765 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:16:59.803 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:16:59.804 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:16:59.805 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:16:59.833 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:16:59.871 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T15:16:59.872 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T15:16:59.907 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:17:00.854 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:19:00.903 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:19:00.943 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:19:02.022 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:19:02.040 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:19:02.040 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:19:02.040 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:19:02.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:19:02.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:19:02.291 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:19:02.499 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:19:02.530 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:19:02.530 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:19:02.567 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:19:02.567 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:19:02.569 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:19:02.602 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:19:02.636 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T15:19:02.637 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T15:19:02.672 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:19:03.708 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:21:03.766 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:21:03.805 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:21:04.840 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:21:04.857 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:21:04.857 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:21:04.858 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:21:04.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:21:05.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:21:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:21:39.778 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T15:21:39.813 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:21:40.821 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:23:40.872 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:23:40.910 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:23:41.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:23:42.014 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:23:42.014 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:23:42.014 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:23:42.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:23:42.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:23:42.277 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:23:45.286 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T15:23:45.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:23:46.359 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:25:46.412 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:25:46.449 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:25:47.582 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:25:47.600 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:25:47.600 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:25:47.600 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:25:47.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:25:47.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:25:47.849 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:25:47.921 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:25:48.980 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:25:50.076 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:25:51.134 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:25:52.233 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:25:53.333 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:25:54.429 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:25:55.528 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:25:56.623 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:25:57.719 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:25:57.720 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T15:25:57.757 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:25:58.800 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:27:58.855 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:27:58.896 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:28:00.019 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:28:00.037 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:28:00.037 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:28:00.037 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:28:00.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:28:00.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:28:00.257 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T15:28:00.295 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:28:01.369 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:30:01.425 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:30:01.462 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:30:02.489 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:30:02.507 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:30:02.508 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:30:02.508 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:30:02.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:30:02.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:30:02.766 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:30:02.912 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T15:30:02.951 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:30:02.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:30:02.997 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:30:02.997 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:30:03.000 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:30:03.018 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T15:30:03.026 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:30:03.094 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:30:03.094 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:30:03.111 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:30:03.111 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:30:03.113 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:30:03.161 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:30:03.171 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T15:30:03.228 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:30:03.228 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:30:03.244 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:30:03.244 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:30:03.320 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:30:04.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:32:04.406 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:32:04.445 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:32:05.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:32:05.512 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:32:05.513 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:32:05.513 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:32:05.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:32:05.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:32:05.760 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:32:24.134 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T15:32:24.174 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:32:25.203 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:34:25.267 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:34:25.310 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:34:26.377 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:34:26.408 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:34:26.409 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:34:26.409 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:34:26.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:34:26.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:34:26.658 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:34:26.811 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T15:34:26.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:34:26.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:34:26.881 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:34:26.881 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:34:26.883 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:34:26.909 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-03-21T15:34:26.910 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:34:26.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:34:26.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:34:27.009 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:34:27.046 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:34:27.053 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T15:34:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:34:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:34:27.129 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:34:27.129 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:34:27.168 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:34:28.267 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:36:28.322 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:36:28.360 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:36:29.436 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:36:29.453 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:36:29.453 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:36:29.453 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:36:29.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:36:29.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:36:29.715 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:36:29.854 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T15:36:29.884 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:36:29.884 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:36:29.901 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:36:29.902 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:36:29.904 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:36:29.952 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:36:30.020 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:36:30.020 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:36:30.065 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:36:30.091 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T15:36:30.097 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:36:30.164 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:36:30.165 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:36:30.183 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:36:30.184 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:36:30.222 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:36:31.292 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:38:31.346 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:38:31.383 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:38:32.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:38:32.379 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:38:32.379 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:38:32.379 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:38:32.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:38:32.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:38:32.625 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:38:32.794 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T15:38:32.831 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:38:32.831 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:38:32.849 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:38:32.849 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:38:32.851 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:38:32.903 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:38:32.969 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:38:32.970 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:38:33.012 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:38:33.044 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:38:33.057 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T15:38:33.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:38:33.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:38:33.128 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:38:33.128 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:38:33.171 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:38:34.240 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:40:34.292 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:40:34.327 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:40:35.406 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:40:35.423 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:40:35.423 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:40:35.424 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:40:35.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:40:35.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:40:35.669 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:40:35.821 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T15:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:40:35.869 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:40:35.869 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:40:35.871 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:40:35.923 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:40:35.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:40:35.994 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:40:36.033 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:40:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:40:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:40:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:40:36.214 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:40:37.225 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:42:37.279 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:42:37.347 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:42:38.404 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:42:38.421 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:42:38.421 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:42:38.421 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:42:38.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:42:38.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:42:38.684 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:42:38.816 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T15:42:38.853 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:42:38.853 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:42:38.874 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:42:38.875 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:42:38.877 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:42:38.921 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:42:38.990 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:42:38.990 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:42:39.033 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:42:39.057 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T15:42:39.057 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/4dcc2052-8a5b-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T15:42:39.067 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:42:39.079 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T15:42:39.117 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:42:40.090 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:44:40.140 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:44:40.180 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:44:41.278 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:44:41.296 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:44:41.296 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:44:41.296 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:44:41.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:44:41.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:44:41.553 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:44:41.740 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T15:44:41.770 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:44:41.770 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:44:41.786 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:44:41.786 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:44:41.788 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:44:41.839 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:44:41.908 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:44:41.908 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:44:41.951 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:44:41.970 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T15:44:41.970 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/4dcc2052-8a5b-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T15:44:41.977 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:44:42.192 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/4dcc2052-8a5b-11eb-90a1-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2021-03-21T15:44:42.194 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T15:44:42.261 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:44:43.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:46:43.339 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:46:43.378 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:46:44.409 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:46:44.428 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:46:44.428 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:46:44.428 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:46:44.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:46:44.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:46:44.675 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:46:44.825 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T15:46:44.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:46:44.857 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:46:44.874 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:46:44.874 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:46:44.877 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:46:44.926 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:46:44.994 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:46:44.995 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:46:45.035 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:46:45.060 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T15:46:45.060 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/4dcc2052-8a5b-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T15:46:45.063 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:46:45.065 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T15:46:45.101 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:46:46.084 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:48:46.138 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:48:46.177 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:48:47.230 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:48:47.247 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:48:47.247 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:48:47.247 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:48:47.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:48:47.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:48:47.513 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:49:05.858 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T15:49:05.895 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:49:06.975 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:51:07.030 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:51:07.100 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:51:08.188 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:51:08.216 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:51:08.216 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:51:08.216 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:51:08.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:51:08.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:51:08.473 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:51:08.545 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:51:09.602 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:51:10.698 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:51:11.761 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:51:12.861 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:51:13.958 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:51:15.018 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:51:16.115 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:51:17.172 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:51:18.269 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T15:51:18.269 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T15:51:18.307 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:51:19.320 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:53:19.373 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:53:19.410 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:53:20.396 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:53:20.414 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:53:20.414 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:53:20.414 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:53:20.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:53:20.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:53:20.661 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:54:20.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T15:54:20.686 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:54:21.674 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:56:21.729 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:56:21.768 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:56:22.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:56:22.840 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:56:22.841 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:56:22.841 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:56:22.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:56:23.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:56:23.110 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:56:23.394 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T15:56:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:56:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:56:23.453 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T15:56:23.453 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:56:23.458 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:56:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T15:56:23.513 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T15:56:23.567 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:56:23.567 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:56:23.581 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T15:56:23.582 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:56:23.584 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:56:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T15:56:23.642 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T15:56:23.707 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:56:23.708 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:56:23.724 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T15:56:23.724 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T15:56:23.763 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:56:24.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:58:24.780 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T15:58:24.821 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:58:25.877 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T15:58:25.895 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T15:58:25.895 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T15:58:25.895 INFO:teuthology.misc:Compressing logs... 2021-03-21T15:58:25.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:58:26.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T15:58:26.147 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T15:58:44.674 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T15:58:44.711 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T15:58:45.695 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:00:45.760 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:00:45.805 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:00:46.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:00:46.874 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:00:46.874 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:00:46.874 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:00:46.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:00:47.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:00:47.132 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:00:47.422 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:00:47.452 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:00:47.452 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:00:47.481 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:00:47.482 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:00:47.483 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:00:47.524 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:00:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:00:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:00:47.638 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:00:47.668 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:00:47.694 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T16:00:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:00:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:00:47.753 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:00:47.753 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:00:47.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:00:48.858 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:02:48.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:02:48.950 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:02:49.994 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:02:50.012 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:02:50.013 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:02:50.013 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:02:50.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:02:50.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:02:50.277 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:02:50.563 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:02:50.592 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:02:50.592 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:02:50.621 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:02:50.621 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:02:50.623 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:02:50.661 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:02:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:02:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:02:50.777 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:02:50.803 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:02:50.829 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T16:02:50.871 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:02:50.871 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:02:50.889 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:02:50.889 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:02:50.926 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:02:51.965 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:04:52.017 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:04:52.062 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:04:53.143 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:04:53.159 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:04:53.160 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:04:53.160 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:04:53.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:04:53.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:04:53.415 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:04:53.719 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:04:53.752 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:04:53.752 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:04:53.780 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:04:53.780 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:04:53.781 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:04:53.823 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:04:53.893 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:04:53.893 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:04:53.929 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:04:53.962 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:04:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:04:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:04:54.086 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:04:55.038 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:06:55.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:06:55.177 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:06:56.225 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:06:56.242 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:06:56.242 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:06:56.243 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:06:56.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:06:56.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:06:56.490 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:06:56.782 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:06:56.813 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:06:56.813 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:06:56.841 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:06:56.841 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:06:56.843 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:06:56.883 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:06:56.953 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:06:56.953 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:06:56.996 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:06:57.023 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:06:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:06:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:06:57.159 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:06:58.177 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:08:58.230 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:08:58.267 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:08:59.287 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:08:59.304 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:08:59.304 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:08:59.304 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:08:59.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:08:59.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:08:59.571 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:08:59.862 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:08:59.893 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:08:59.894 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:08:59.922 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:08:59.922 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:08:59.925 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:08:59.962 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:09:00.032 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:09:00.032 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:09:00.070 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:09:00.100 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:09:00.125 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T16:09:00.155 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:09:00.189 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:09:01.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:11:01.235 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:11:01.273 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:11:02.281 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:11:02.311 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:11:02.311 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:11:02.311 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:11:02.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:11:02.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:11:02.561 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:11:02.857 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:11:02.887 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:11:02.887 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:11:02.917 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:11:02.917 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:11:02.920 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:11:02.957 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:11:03.026 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:11:03.026 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:11:03.070 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:11:03.096 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:11:03.123 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T16:11:03.127 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:11:03.164 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:11:04.156 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:13:04.210 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:13:04.248 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:13:05.355 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:13:05.372 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:13:05.372 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:13:05.372 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:13:05.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:13:05.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:13:05.619 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:13:05.922 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:13:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:13:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:13:05.981 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:13:05.981 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:13:05.984 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:13:06.026 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:13:06.096 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:13:06.096 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:13:06.132 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:13:06.165 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:13:06.188 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T16:13:06.252 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/17313736-8a5f-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T16:13:06.269 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:13:06.308 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:13:07.363 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:15:07.416 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:15:07.455 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:15:08.536 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:15:08.553 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:15:08.553 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:15:08.553 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:15:08.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:15:08.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:15:08.821 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:15:09.108 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:15:09.138 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:15:09.138 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:15:09.169 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:15:09.170 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:15:09.171 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:15:09.209 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:15:09.279 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:15:09.279 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:15:09.318 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:15:09.348 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:15:09.375 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T16:15:09.376 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/17313736-8a5f-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T16:15:09.377 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:15:09.414 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:15:10.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:17:10.545 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:17:10.582 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:17:12.337 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:17:12.353 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:17:12.353 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:17:12.354 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:17:12.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:17:12.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:17:12.604 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:17:12.904 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:17:12.936 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:17:12.936 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:17:12.961 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:17:12.961 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:17:12.963 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:17:13.007 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:17:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:17:13.077 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:17:13.119 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:17:13.146 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:17:13.173 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T16:17:13.173 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/17313736-8a5f-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T16:17:13.175 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:17:13.214 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:17:14.240 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:19:14.292 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:19:14.357 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:19:15.432 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:19:15.449 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:19:15.449 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:19:15.449 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:19:15.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:19:15.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:19:15.702 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:19:16.001 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:19:16.033 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:19:16.033 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:19:16.060 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:19:16.060 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:19:16.064 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:19:16.102 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:19:16.170 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:19:16.171 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:19:16.209 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:19:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:19:16.266 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T16:19:16.266 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/17313736-8a5f-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T16:19:16.269 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:19:16.308 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:19:17.342 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:21:17.392 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:21:17.431 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:21:18.431 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:21:18.446 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:21:18.446 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:21:18.447 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:21:18.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:21:18.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:21:18.711 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:21:18.995 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:21:19.026 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:21:19.027 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:21:19.055 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:21:19.055 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:21:19.059 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:21:19.097 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:21:19.168 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:21:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:21:19.201 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:21:19.237 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:21:19.258 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T16:21:19.258 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/17313736-8a5f-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T16:21:19.261 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:21:19.295 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:21:20.249 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:23:20.297 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:23:20.356 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:23:22.208 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:23:22.226 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:23:22.226 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:23:22.226 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:23:22.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:23:22.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:23:22.476 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:24:22.450 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T16:24:22.496 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:24:24.315 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:26:24.368 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:26:24.409 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:26:26.419 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:26:26.436 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:26:26.436 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:26:26.437 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:26:26.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:26:26.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:26:26.705 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:26:26.759 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T16:26:27.816 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T16:26:28.874 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T16:26:29.976 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T16:26:31.074 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T16:26:32.173 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T16:26:33.235 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T16:26:34.335 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T16:26:35.440 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T16:26:36.538 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T16:26:36.538 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T16:26:36.580 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:26:38.470 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:28:38.523 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:28:38.566 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:28:40.393 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:28:40.419 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:28:40.420 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:28:40.420 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:28:40.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:28:40.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:28:40.676 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:28:41.019 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:28:41.052 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:28:41.052 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:28:41.079 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:28:41.079 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:28:41.080 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:28:41.123 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:28:41.141 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T16:28:41.192 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:28:41.193 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:28:41.209 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:28:41.209 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:28:41.212 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:28:41.260 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:28:41.266 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T16:28:41.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:28:41.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:28:41.347 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:28:41.347 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:28:41.385 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:28:42.459 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:30:42.515 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:30:42.588 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:30:44.782 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:30:44.800 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:30:44.800 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:30:44.800 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:30:44.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:30:45.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:30:45.054 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:30:45.396 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:30:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:30:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:30:45.454 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:30:45.454 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:30:45.457 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:30:45.501 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:30:45.515 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T16:30:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:30:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:30:45.589 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:30:45.590 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:30:45.591 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:30:45.643 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:30:45.644 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T16:30:45.714 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:30:45.715 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:30:45.737 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:30:45.737 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:30:45.787 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:30:48.248 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:32:48.307 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:32:48.363 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:32:50.407 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:32:50.425 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:32:50.426 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:32:50.426 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:32:50.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:32:50.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:32:50.693 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:33:22.798 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:33:22.838 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:33:22.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:33:22.855 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:33:22.855 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:33:22.857 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:33:22.911 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:33:22.917 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T16:33:22.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:33:22.980 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:33:22.997 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:33:22.997 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:33:22.998 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:33:23.048 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:33:23.053 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T16:33:23.119 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:33:23.119 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:33:23.136 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:33:23.136 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:33:23.185 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:33:25.038 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:35:25.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:35:25.157 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:35:27.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:35:27.087 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:35:27.087 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:35:27.087 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:35:27.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:35:27.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:35:27.340 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:35:27.635 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T16:35:27.667 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:35:27.667 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:35:27.695 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:35:27.695 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:35:27.697 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:35:27.737 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:35:27.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:35:27.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:35:27.845 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:35:27.876 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:35:27.895 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T16:35:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:35:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:35:27.961 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:35:27.961 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:35:28.003 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:35:29.959 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:37:30.014 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:37:30.053 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:37:32.077 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:37:32.094 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:37:32.094 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:37:32.094 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:37:32.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:37:32.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:37:32.347 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:37:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:37:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:37:32.683 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:37:32.725 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:37:32.796 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:37:32.796 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:37:32.838 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:37:32.871 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:37:32.940 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:37:32.940 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:37:33.037 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:37:34.942 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:39:34.997 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:39:35.042 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:39:36.928 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:39:36.950 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:39:36.950 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:39:36.951 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:39:36.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:39:37.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:39:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:39:37.560 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:39:37.560 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:39:37.592 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:39:37.629 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:39:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:39:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:39:37.735 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:39:37.767 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:39:37.854 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T16:39:37.854 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:39:37.924 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:39:38.983 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:41:39.043 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:41:39.157 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:41:40.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:41:40.224 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:41:40.224 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:41:40.224 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:41:40.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:41:40.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:41:40.478 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:41:40.817 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:41:40.817 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:41:40.848 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:41:40.886 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:41:40.956 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:41:40.956 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:41:40.991 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:41:41.025 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:41:41.282 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T16:41:41.286 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T16:41:41.316 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:41:41.356 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:41:42.414 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:43:42.465 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:43:42.507 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:43:43.610 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:43:43.628 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:43:43.628 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:43:43.628 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:43:43.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:43:43.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:43:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:43:44.220 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:43:44.220 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:43:44.259 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:43:44.291 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:43:44.362 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:43:44.362 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:43:44.402 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:43:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:43:44.658 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T16:43:44.666 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T16:43:45.327 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:43:45.369 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:43:46.453 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:45:46.521 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:45:46.563 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:45:47.589 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:45:47.606 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:45:47.606 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:45:47.606 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:45:47.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:45:47.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:45:47.878 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:45:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:45:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:45:48.235 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:45:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:45:48.343 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:45:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:45:48.381 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:45:48.414 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:45:48.472 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T16:45:48.474 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T16:45:49.041 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:45:49.080 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:45:50.082 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:47:50.137 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:47:50.177 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:47:51.224 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:47:51.242 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:47:51.242 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:47:51.242 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:47:51.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:47:51.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:47:51.496 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:47:51.811 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:47:51.811 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:47:51.843 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:47:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:47:51.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:47:51.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:47:52.000 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:47:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:47:52.606 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T16:47:52.626 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T16:47:52.646 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2-client.mirror.25604.log: file size changed while zipping 2021-03-21T16:47:52.670 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:47:52.716 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:47:53.754 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:49:53.809 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:49:53.849 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:49:54.856 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:49:54.873 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:49:54.873 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:49:54.873 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:49:54.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:49:55.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:49:55.122 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:49:55.448 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:49:55.448 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:49:55.479 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:49:55.518 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:49:55.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:49:55.588 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:49:55.628 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:49:55.655 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:49:56.051 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T16:49:56.093 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T16:49:56.135 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:49:56.173 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:49:57.185 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:51:57.241 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:51:57.283 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:51:58.273 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:51:58.289 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:51:58.289 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:51:58.289 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:51:58.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:51:58.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:51:58.553 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:51:58.880 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:51:58.880 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:51:58.911 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:51:58.951 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:51:59.020 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:51:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:51:59.055 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:51:59.089 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:51:59.713 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T16:51:59.948 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T16:52:00.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:52:00.936 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:52:01.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:54:01.992 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:54:02.044 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:54:03.209 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:54:03.226 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:54:03.226 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:54:03.226 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:54:03.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:54:03.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:54:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:54:03.836 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:54:03.837 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:54:03.865 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:54:03.908 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:54:03.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:54:03.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:54:04.010 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:54:04.050 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:54:04.686 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T16:54:05.130 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T16:54:05.916 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:54:05.953 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:54:07.056 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:56:07.111 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:56:07.150 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:56:08.253 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:56:08.272 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:56:08.272 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:56:08.273 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:56:08.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:56:08.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:56:08.521 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:56:08.858 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:56:08.859 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:56:08.893 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:56:08.929 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:56:08.999 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:56:08.999 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:56:09.041 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:56:09.068 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:56:09.845 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T16:56:10.387 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T16:56:10.736 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:56:10.775 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:56:11.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:58:11.906 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T16:58:11.944 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:58:13.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T16:58:13.224 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T16:58:13.224 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T16:58:13.224 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:58:13.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:58:13.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T16:58:13.505 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T16:58:13.861 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T16:58:13.863 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:58:13.900 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:58:13.939 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T16:58:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T16:58:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T16:58:14.101 INFO:teuthology.misc:Compressing logs... 2021-03-21T16:58:14.144 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T16:58:15.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T16:58:15.754 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T16:58:16.076 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T16:58:16.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T16:58:17.157 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:00:17.216 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:00:17.259 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:00:18.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:00:18.367 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:00:18.367 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:00:18.367 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:00:18.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:00:18.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:00:18.624 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:00:18.968 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:00:18.969 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:00:19.003 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:00:19.040 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:00:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:00:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:00:19.153 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:00:19.182 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:00:20.142 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T17:00:20.147 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T17:00:20.521 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T17:00:20.561 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:00:21.655 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:02:21.714 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:02:21.753 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:02:22.848 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:02:22.865 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:02:22.865 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:02:22.865 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:02:22.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:02:23.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:02:23.125 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:02:23.480 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:02:23.481 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:02:23.507 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:02:23.551 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:02:23.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:02:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:02:23.655 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:02:23.692 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:02:23.740 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T17:02:23.761 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T17:02:23.795 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T17:02:23.839 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:02:24.913 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:04:24.966 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:04:25.003 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:04:26.050 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:04:26.066 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:04:26.066 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:04:26.066 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:04:26.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:04:26.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:04:26.337 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:04:26.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:04:26.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:04:26.694 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:04:26.731 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:04:26.802 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:04:26.802 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:04:26.840 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:04:26.872 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:04:26.943 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T17:04:26.958 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T17:04:26.979 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T17:04:27.016 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:04:28.089 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:06:28.143 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:06:28.188 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:06:29.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:06:29.278 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:06:29.278 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:06:29.278 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:06:29.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:06:29.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:06:29.528 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:06:29.855 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:06:29.855 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:06:29.886 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:06:29.926 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:06:29.998 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:06:29.998 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:06:30.036 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:06:30.066 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:06:30.142 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T17:06:30.163 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T17:06:30.193 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T17:06:30.231 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:06:31.264 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:08:31.316 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:08:31.374 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:08:32.459 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:08:32.481 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:08:32.482 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:08:32.482 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:08:32.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:08:32.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:08:32.736 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:08:33.066 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:08:33.066 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:08:33.098 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:08:33.137 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:08:33.207 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:08:33.207 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:08:33.248 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:08:33.276 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:08:33.427 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T17:08:33.438 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T17:08:33.459 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T17:08:33.506 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:08:34.505 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:10:34.559 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:10:34.602 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:10:35.635 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:10:35.653 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:10:35.653 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:10:35.653 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:10:35.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:10:35.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:10:35.923 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:10:36.247 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:10:36.247 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:10:36.271 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:10:36.318 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:10:36.388 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:10:36.388 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:10:36.436 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:10:36.463 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:10:36.778 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T17:10:37.120 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2-client.mirror.52164.log: file size changed while zipping 2021-03-21T17:10:37.121 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T17:10:37.127 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T17:10:37.164 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:10:38.176 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:12:38.233 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:12:38.275 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:12:39.380 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:12:39.397 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:12:39.397 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:12:39.397 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:12:39.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:12:39.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:12:39.646 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:12:39.998 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:12:39.998 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:12:40.030 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:12:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:12:40.141 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:12:40.141 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:12:40.180 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:12:40.212 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:12:40.409 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T17:12:40.412 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T17:12:40.626 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T17:12:40.686 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:12:41.795 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:14:41.857 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:14:41.898 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:14:42.960 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:14:42.977 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:14:42.978 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:14:42.978 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:14:42.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:14:43.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:14:43.225 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:14:43.557 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:14:43.557 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:14:43.590 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:14:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:14:43.695 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:14:43.695 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:14:43.732 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:14:43.762 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:14:43.803 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T17:14:43.805 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T17:14:44.085 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T17:14:44.176 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:14:45.220 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:16:45.270 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:16:45.311 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:16:46.396 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:16:46.413 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:16:46.413 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:16:46.413 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:16:46.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:16:46.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:16:46.676 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:16:47.006 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:16:47.006 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:16:47.038 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:16:47.079 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:16:47.146 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:16:47.147 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:16:47.188 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:16:47.216 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:16:47.533 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.audit.log.gz already exists; not overwritten 2021-03-21T17:16:47.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cluster2.log.gz already exists; not overwritten 2021-03-21T17:16:47.853 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T17:16:47.891 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:16:48.918 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:18:48.972 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:18:49.009 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:18:50.088 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:18:50.195 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:18:50.195 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:18:50.195 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:18:50.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:18:50.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:18:50.414 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T17:18:50.451 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:18:51.447 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:20:51.501 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:20:51.540 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:20:52.586 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:20:52.603 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:20:52.603 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:20:52.603 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:20:52.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:20:52.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:20:52.859 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:20:55.906 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T17:20:55.944 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:20:56.961 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:22:57.014 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:22:57.054 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:22:58.103 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:22:58.120 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:22:58.120 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:22:58.120 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:22:58.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:22:58.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:22:58.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:22:58.398 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T17:22:59.495 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T17:23:00.593 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T17:24:01.596 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T17:24:01.635 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:24:02.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:26:02.818 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:26:02.869 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:26:04.018 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:26:04.053 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:26:04.054 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:26:04.054 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:26:04.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:26:04.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:26:04.321 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:26:04.662 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T17:26:04.696 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:26:04.696 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:26:04.721 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:26:04.721 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:26:04.724 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:26:04.766 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:26:04.785 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T17:26:04.835 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:26:04.835 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:26:04.851 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:26:04.851 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:26:04.853 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:26:04.903 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:26:04.909 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T17:26:04.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:26:04.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:26:04.990 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:26:04.990 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:26:05.036 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:26:06.111 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:28:06.176 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:28:06.215 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:28:07.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:28:07.285 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:28:07.286 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:28:07.286 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:28:07.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:28:07.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:28:07.550 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:28:07.812 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T17:28:07.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:28:07.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:28:07.867 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:28:07.867 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:28:07.868 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:28:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:28:07.922 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T17:28:07.983 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:28:07.983 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:28:07.999 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:28:07.999 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:28:08.000 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:28:08.051 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:28:08.054 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T17:28:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:28:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:28:08.136 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:28:08.137 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:28:08.173 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:28:09.198 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:30:09.251 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:30:09.286 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:30:10.329 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:30:10.347 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:30:10.347 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:30:10.347 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:30:10.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:30:10.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:30:10.594 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:30:28.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T17:30:28.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:30:29.945 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:32:29.995 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:32:30.033 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:32:31.088 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:32:31.105 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:32:31.105 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:32:31.105 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:32:31.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:32:31.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:32:31.351 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:32:31.642 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T17:32:31.674 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:32:31.674 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:32:31.703 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:32:31.703 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:32:31.706 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:32:31.745 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:32:31.816 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:32:31.816 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:32:31.857 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:32:31.884 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:32:31.912 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T17:32:31.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:32:31.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:32:31.976 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:32:31.976 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:32:32.016 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:32:33.064 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:34:33.115 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:34:33.151 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:34:34.244 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:34:34.260 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:34:34.261 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:34:34.261 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:34:34.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:34:34.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:34:34.528 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:34:34.831 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:34:34.832 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:34:34.858 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:34:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:34:34.973 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:34:34.973 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:34:35.015 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:34:35.042 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:34:35.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:34:35.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:34:35.169 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:34:36.154 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:36:36.206 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:36:36.244 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:36:37.270 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:36:37.286 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:36:37.286 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:36:37.287 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:36:37.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:36:37.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:36:37.537 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:36:37.869 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:36:37.870 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:36:37.902 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:36:37.939 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:36:38.009 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:36:38.009 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:36:38.050 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:36:38.077 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:36:38.144 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:36:38.144 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:36:38.199 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:36:39.188 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:38:39.243 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:38:39.281 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:38:40.291 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:38:40.312 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:38:40.312 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:38:40.312 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:38:40.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:38:40.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:38:40.558 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:38:40.886 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:38:40.886 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:38:40.918 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:38:40.957 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:38:41.026 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:38:41.026 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:38:41.070 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:38:41.097 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:38:41.164 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:38:41.165 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:38:41.231 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:38:42.184 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:40:42.242 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:40:42.280 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:40:43.284 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:40:43.300 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:40:43.300 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:40:43.300 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:40:43.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:40:43.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:40:43.561 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:40:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:40:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:40:43.914 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:40:43.955 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:40:44.023 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:40:44.024 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:40:44.091 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:40:44.119 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:40:44.187 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:40:44.187 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:40:44.403 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:40:45.458 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:42:45.512 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:42:45.552 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:42:46.597 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:42:46.614 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:42:46.614 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:42:46.615 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:42:46.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:42:46.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:42:46.863 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:42:47.204 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:42:47.204 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:42:47.238 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:42:47.274 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:42:47.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:42:47.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:42:47.387 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:42:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:42:47.482 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:42:47.483 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:42:47.663 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:42:48.741 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:44:48.805 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:44:48.847 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:44:49.890 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:44:49.907 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:44:49.907 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:44:49.907 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:44:49.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:44:50.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:44:50.168 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:44:50.511 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:44:50.511 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:44:50.539 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:44:50.582 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:44:50.655 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:44:50.655 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:44:50.706 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:44:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:44:50.804 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:44:50.805 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:44:51.037 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:44:52.088 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:46:52.145 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:46:52.192 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:46:53.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:46:53.249 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:46:53.249 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:46:53.250 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:46:53.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:46:53.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:46:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:46:53.894 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:46:53.894 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:46:53.923 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:46:53.968 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:46:54.040 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:46:54.041 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:46:54.091 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:46:54.127 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:46:54.201 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:46:54.201 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:46:54.394 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:46:55.400 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:48:55.454 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:48:55.489 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:48:56.523 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:48:56.541 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:48:56.541 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:48:56.541 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:48:56.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:48:56.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:48:56.795 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:48:57.140 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:48:57.140 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:48:57.170 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:48:57.211 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:48:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:48:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:48:57.334 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:48:57.363 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:48:57.433 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:48:57.433 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:48:57.654 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:48:58.690 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:50:58.744 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:50:58.784 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:50:59.809 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:50:59.826 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:50:59.826 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:50:59.827 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:50:59.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:51:00.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:51:00.077 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:51:00.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:51:00.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:51:00.455 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:51:00.495 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:51:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:51:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:51:00.609 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:51:00.637 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:51:00.707 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:51:00.708 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:51:00.930 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:51:01.970 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:53:02.020 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:53:02.059 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:53:03.159 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:53:03.176 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:53:03.177 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:53:03.177 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:53:03.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:53:03.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:53:03.446 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:53:03.825 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T17:53:03.825 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:53:03.850 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:53:03.899 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T17:53:03.907 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T17:53:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:53:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:53:03.983 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T17:53:03.983 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:53:03.985 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:53:04.037 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T17:53:04.107 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T17:53:04.107 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T17:53:04.411 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:53:05.473 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:55:05.526 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:55:05.566 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:55:06.550 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:55:06.567 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:55:06.567 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:55:06.567 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:55:06.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:55:06.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:55:06.814 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:55:25.250 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T17:55:25.285 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:55:26.245 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:57:26.297 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T17:57:26.335 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:57:27.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T17:57:27.353 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T17:57:27.353 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T17:57:27.354 INFO:teuthology.misc:Compressing logs... 2021-03-21T17:57:27.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:57:27.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T17:57:27.602 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T17:57:27.633 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T17:57:28.729 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T17:57:29.826 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T17:57:30.923 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T17:57:32.021 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T17:57:33.117 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T17:57:34.174 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T17:57:35.233 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T17:58:10.887 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T17:58:10.927 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T17:58:11.961 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:00:12.013 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:00:12.074 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:00:13.130 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:00:13.147 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:00:13.148 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:00:13.148 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:00:13.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:00:13.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:00:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:00:13.699 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T18:00:13.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:00:13.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:00:13.759 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:00:13.759 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:00:13.762 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:00:13.800 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:00:13.819 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T18:00:13.870 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:00:13.870 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:00:13.887 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:00:13.887 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:00:13.889 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:00:13.938 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:00:13.947 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T18:00:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:00:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:00:14.022 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:00:14.022 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:00:14.062 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:00:15.149 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:02:15.223 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:02:15.260 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:02:16.273 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:02:16.290 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:02:16.290 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:02:16.290 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:02:16.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:02:16.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:02:16.541 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:02:35.079 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T18:02:35.117 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:02:36.096 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:04:36.148 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:04:36.187 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:04:37.211 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:04:37.228 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:04:37.228 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:04:37.228 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:04:37.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:04:37.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:04:37.479 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:04:37.794 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T18:04:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:04:37.826 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:04:37.854 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:04:37.854 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:04:37.856 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:04:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:04:37.965 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:04:37.965 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:04:38.011 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:04:38.038 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:04:38.068 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T18:04:38.107 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:04:38.107 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:04:38.125 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:04:38.126 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:04:38.164 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:04:39.199 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:06:39.253 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:06:39.293 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:06:40.324 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:06:40.341 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:06:40.341 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:06:40.341 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:06:40.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:06:40.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:06:40.605 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:06:40.896 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:06:40.896 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:06:40.929 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:06:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:06:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:06:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:06:41.076 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:06:41.108 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:06:41.176 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:06:41.176 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:06:41.228 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:06:42.194 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:08:42.246 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:08:42.284 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:08:43.304 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:08:43.321 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:08:43.321 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:08:43.321 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:08:43.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:08:43.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:08:43.570 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:08:43.900 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:08:43.900 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:08:43.935 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:08:43.971 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:08:44.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:08:44.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:08:44.082 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:08:44.110 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:08:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:08:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:08:44.337 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:08:45.315 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:10:45.367 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:10:45.402 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:10:46.418 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:10:46.433 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:10:46.433 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:10:46.433 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:10:46.433 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:10:46.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:10:46.685 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:10:47.031 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:10:47.031 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:10:47.060 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:10:47.102 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:10:47.172 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:10:47.172 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:10:47.213 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:10:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:10:47.269 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-21T18:10:47.269 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-21T18:10:47.270 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T18:10:47.309 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:10:48.304 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:12:48.354 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:12:48.392 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:12:49.323 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:12:49.339 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:12:49.339 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:12:49.339 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:12:49.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:12:49.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:12:49.603 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:12:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:12:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:12:49.959 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:12:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:12:50.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:12:50.060 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:12:50.103 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:12:50.130 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:12:50.158 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-21T18:12:50.158 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-21T18:12:50.159 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T18:12:50.193 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:12:51.130 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:14:51.185 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:14:51.223 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:14:52.257 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:14:52.274 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:14:52.274 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:14:52.274 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:14:52.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:14:52.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:14:52.529 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:14:52.870 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:14:52.870 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:14:52.903 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:14:52.941 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:14:53.010 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:14:53.010 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:14:53.052 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:14:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:14:53.107 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-21T18:14:53.107 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-21T18:14:53.109 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T18:14:53.148 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:14:54.164 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:16:54.217 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:16:54.253 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:16:55.237 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:16:55.253 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:16:55.253 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:16:55.253 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:16:55.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:16:55.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:16:55.500 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:16:55.823 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:16:55.823 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:16:55.853 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:16:55.892 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:16:55.961 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:16:55.961 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:16:56.002 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:16:56.031 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:16:56.058 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-21T18:16:56.059 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-21T18:16:56.060 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T18:16:56.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:16:57.079 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:18:57.132 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:18:57.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:18:58.118 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:18:58.134 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:18:58.135 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:18:58.135 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:18:58.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:18:58.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:18:58.395 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:18:58.722 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:18:58.722 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:18:58.757 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:18:58.790 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:18:58.858 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:18:58.859 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:18:58.898 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:18:58.926 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:18:58.950 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-21T18:18:58.950 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-21T18:18:58.953 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T18:18:58.986 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:18:59.927 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:20:59.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:21:00.024 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:21:00.967 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:21:00.982 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:21:00.983 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:21:00.983 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:21:00.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:21:01.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:21:01.228 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:21:19.682 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T18:21:19.716 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:21:20.660 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:23:20.716 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:23:20.751 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:23:21.807 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:23:21.823 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:23:21.824 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:23:21.824 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:23:21.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:23:22.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:23:22.078 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:23:40.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T18:23:40.516 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:23:41.516 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:25:41.573 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:25:41.614 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:25:42.669 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:25:42.686 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:25:42.686 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:25:42.686 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:25:42.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:25:42.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:25:42.952 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:25:44.514 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T18:25:44.551 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:25:45.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:27:45.577 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:27:45.616 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:27:46.662 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:27:46.679 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:27:46.679 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:27:46.679 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:27:46.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:27:46.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:27:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:27:47.233 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T18:27:47.264 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:27:47.264 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:27:47.294 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:27:47.294 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:27:47.295 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:27:47.335 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:27:47.352 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T18:27:47.405 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:27:47.405 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:27:47.421 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:27:47.421 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:27:47.423 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:27:47.474 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:27:47.480 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T18:27:47.543 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:27:47.543 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:27:47.561 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:27:47.561 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:27:47.647 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:27:48.724 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:29:48.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:29:48.812 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:29:49.823 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:29:49.840 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:29:49.840 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:29:49.840 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:29:49.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:29:50.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:29:50.093 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:29:50.376 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T18:29:50.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:29:50.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:29:50.437 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:29:50.437 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:29:50.440 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:29:50.477 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:29:50.497 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T18:29:50.547 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:29:50.547 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:29:50.564 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:29:50.565 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:29:50.566 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:29:50.617 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:29:50.622 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T18:29:50.686 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:29:50.686 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:29:50.701 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:29:50.701 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:29:50.739 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:29:51.731 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:31:51.798 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:31:51.833 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:31:52.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:31:52.815 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:31:52.815 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:31:52.815 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:31:52.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:31:53.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:31:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:32:11.458 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T18:32:11.493 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:32:12.450 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:34:12.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:34:12.542 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:34:13.523 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:34:13.540 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:34:13.540 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:34:13.540 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:34:13.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:34:13.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:34:13.790 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:34:14.074 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T18:34:14.106 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:34:14.106 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:34:14.134 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:34:14.134 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:34:14.136 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:34:14.183 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:34:14.255 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:34:14.255 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:34:14.294 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:34:14.325 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:34:14.350 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T18:34:14.395 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:34:14.395 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:34:14.409 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:34:14.409 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:34:14.449 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:34:15.488 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:36:15.539 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:36:15.577 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:36:16.570 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:36:16.587 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:36:16.588 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:36:16.588 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:36:16.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:36:16.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:36:16.836 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:36:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:36:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:36:17.169 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:36:17.212 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:36:17.281 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:36:17.281 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:36:17.323 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:36:17.356 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:36:17.427 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:36:17.427 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:36:17.479 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:36:18.465 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:38:18.518 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:38:18.562 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:38:19.629 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:38:19.650 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:38:19.650 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:38:19.650 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:38:19.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:38:19.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:38:19.917 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:38:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:38:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:38:20.276 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:38:20.316 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:38:20.387 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:38:20.387 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:38:20.427 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:38:20.456 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:38:20.549 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T18:38:20.550 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:38:20.627 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:38:21.758 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:40:21.812 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:40:21.851 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:40:22.870 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:40:22.904 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:40:22.904 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:40:22.904 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:40:22.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:40:23.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:40:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:40:23.504 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:40:23.504 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:40:23.533 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:40:23.574 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:40:23.645 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:40:23.645 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:40:23.682 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:40:23.714 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:40:23.738 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-21T18:40:23.740 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T18:40:23.781 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:40:24.785 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:42:24.839 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:42:24.879 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:42:25.847 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:42:25.866 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:42:25.867 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:42:25.867 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:42:25.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:42:26.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:42:26.114 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:42:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:42:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:42:26.491 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:42:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:42:26.597 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:42:26.597 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:42:26.637 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:42:26.666 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:42:26.692 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-21T18:42:26.695 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T18:42:26.728 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:42:27.731 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:44:27.787 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:44:27.827 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:44:28.874 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:44:28.893 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:44:28.894 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:44:28.894 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:44:28.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:44:29.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:44:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:44:29.499 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:44:29.499 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:44:29.531 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:44:29.570 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:44:29.641 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:44:29.641 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:44:29.683 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:44:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:44:29.736 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-21T18:44:29.738 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T18:44:29.776 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:44:30.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:46:30.859 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:46:30.896 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:46:31.896 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:46:31.912 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:46:31.912 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:46:31.912 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:46:31.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:46:32.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:46:32.163 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:46:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:46:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:46:32.531 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:46:32.572 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:46:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:46:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:46:32.673 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:46:32.713 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:46:32.732 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-21T18:46:32.732 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T18:46:32.769 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:46:33.751 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:48:33.803 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:48:33.842 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:48:34.829 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:48:34.846 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:48:34.846 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:48:34.847 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:48:34.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:48:35.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:48:35.092 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:48:53.442 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T18:48:53.479 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:48:54.441 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:50:54.493 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:50:54.528 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:50:55.546 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:50:55.562 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:50:55.562 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:50:55.562 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:50:55.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:50:55.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:50:55.825 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:50:56.158 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:50:56.158 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:50:56.190 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:50:56.228 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:50:56.299 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:50:56.299 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:50:56.351 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:50:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:50:56.415 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-21T18:50:56.417 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T18:50:56.453 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:50:57.456 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:52:57.509 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:52:57.546 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:52:58.578 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:52:58.596 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:52:58.596 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:52:58.596 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:52:58.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:52:58.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:52:58.847 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:53:14.886 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T18:53:14.886 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:53:14.915 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:53:14.958 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T18:53:15.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T18:53:15.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T18:53:15.086 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:53:15.114 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T18:53:15.168 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-21T18:53:15.169 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T18:53:15.207 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:53:16.184 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:55:16.255 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:55:16.293 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:55:17.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:55:17.352 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:55:17.352 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:55:17.352 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:55:17.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:55:17.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:55:17.572 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T18:55:17.606 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:55:18.620 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:57:18.679 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:57:18.727 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:57:19.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:57:19.887 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:57:19.887 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:57:19.887 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:57:19.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:57:20.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:57:20.156 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:57:38.498 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T18:57:38.537 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:57:39.593 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:59:39.651 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T18:59:39.689 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:59:40.792 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T18:59:40.809 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T18:59:40.809 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T18:59:40.809 INFO:teuthology.misc:Compressing logs... 2021-03-21T18:59:40.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:59:41.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T18:59:41.065 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T18:59:41.095 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T18:59:42.154 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T18:59:43.251 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T18:59:44.353 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T18:59:45.452 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T18:59:46.511 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T18:59:47.607 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T18:59:48.669 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T18:59:49.769 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T18:59:50.866 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T18:59:50.867 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T18:59:50.906 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T18:59:51.931 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:01:51.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:01:52.021 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:01:54.088 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:01:54.110 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:01:54.111 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:01:54.111 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:01:54.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:01:54.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:01:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:02:54.333 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T19:02:54.371 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:02:55.383 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:04:55.436 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:04:55.473 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:04:56.505 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:04:56.573 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:04:56.574 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:04:56.574 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:04:56.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:04:56.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:04:56.830 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:04:57.085 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T19:04:57.119 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:04:57.119 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:04:57.140 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:04:57.140 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:04:57.143 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:04:57.187 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:04:57.197 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T19:04:57.254 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:04:57.254 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:04:57.268 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:04:57.268 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:04:57.270 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:04:57.321 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:04:57.468 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T19:04:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:04:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:04:57.513 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:04:57.513 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:04:57.553 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:04:58.581 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:06:58.631 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:06:58.669 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:06:59.686 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:06:59.703 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:06:59.703 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:06:59.703 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:06:59.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:06:59.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:06:59.920 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T19:06:59.959 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:07:00.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:09:01.030 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:09:01.070 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:09:02.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:09:02.209 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:09:02.209 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:09:02.209 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:09:02.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:09:02.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:09:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:09:02.754 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T19:09:02.785 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:09:02.785 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:09:02.811 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:09:02.811 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:09:02.813 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:09:02.857 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:09:02.928 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:09:02.928 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:09:02.962 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:09:02.998 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:09:03.010 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T19:09:03.068 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:09:03.068 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:09:03.083 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:09:03.083 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:09:03.122 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:09:04.202 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:11:04.257 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:11:04.296 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:11:05.294 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:11:05.313 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:11:05.313 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:11:05.313 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:11:05.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:11:05.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:11:05.575 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:11:05.879 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:11:05.880 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:11:05.910 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:11:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:11:06.022 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:11:06.022 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:11:06.070 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:11:06.097 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:11:06.169 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:11:06.169 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:11:06.226 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:11:07.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:13:07.291 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:13:07.329 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:13:08.359 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:13:08.376 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:13:08.376 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:13:08.376 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:13:08.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:13:08.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:13:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:13:08.963 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:13:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:13:08.995 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:13:09.033 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:13:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:13:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:13:09.140 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:13:09.170 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:13:09.238 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:13:09.238 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:13:09.298 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:13:10.421 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:15:10.477 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:15:10.516 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:15:11.552 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:15:11.569 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:15:11.569 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:15:11.569 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:15:11.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:15:11.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:15:11.816 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:15:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:15:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:15:13.196 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:15:13.236 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:15:13.307 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:15:13.307 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:15:13.343 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:15:13.375 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:15:13.445 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:15:13.445 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:15:13.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:15:14.599 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:17:14.653 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:17:14.809 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:17:16.049 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:17:16.068 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:17:16.068 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:17:16.068 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:17:16.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:17:16.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:17:16.332 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:17:16.657 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:17:16.657 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:17:16.690 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:17:16.727 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:17:16.797 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:17:16.797 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:17:16.832 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:17:16.866 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:17:16.889 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T19:17:16.928 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T19:17:16.988 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:17:18.013 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:19:18.069 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:19:18.107 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:19:19.167 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:19:19.184 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:19:19.184 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:19:19.184 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:19:19.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:19:19.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:19:19.436 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:19:19.758 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:19:19.758 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:19:19.794 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:19:19.830 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:19:19.901 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:19:19.901 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:19:19.944 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:19:19.972 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:19:20.002 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T19:19:20.002 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/59dde6b4-8a79-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T19:19:20.004 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T19:19:20.043 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:19:21.105 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:21:21.159 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:21:21.193 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:21:22.219 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:21:22.235 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:21:22.235 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:21:22.235 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:21:22.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:21:22.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:21:22.484 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:21:22.825 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:21:22.825 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:21:22.856 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:21:22.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:21:22.966 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:21:22.966 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:21:23.002 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:21:23.035 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:21:23.059 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T19:21:23.079 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/59dde6b4-8a79-11eb-90a1-001a4aab830c/ceph-osd.11.log: file size changed while zipping 2021-03-21T19:21:23.128 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/59dde6b4-8a79-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T19:21:23.158 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T19:21:23.195 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:21:24.189 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:23:24.241 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:23:24.281 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:23:25.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:23:25.378 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:23:25.379 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:23:25.379 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:23:25.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:23:25.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:23:25.658 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:23:25.988 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:23:25.989 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:23:26.024 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:23:26.060 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:23:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:23:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:23:26.165 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:23:26.200 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:23:26.222 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T19:23:26.222 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/59dde6b4-8a79-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T19:23:26.224 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T19:23:26.261 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:23:27.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:25:27.393 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:25:27.432 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:25:28.490 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:25:28.508 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:25:28.508 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:25:28.508 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:25:28.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:25:28.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:25:28.760 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:25:29.103 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:25:29.103 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:25:29.134 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:25:29.173 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:25:29.244 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:25:29.244 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:25:29.298 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:25:29.328 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:25:29.354 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T19:25:29.355 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/59dde6b4-8a79-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T19:25:29.356 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T19:25:29.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:25:30.447 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:27:30.498 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:27:30.535 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:27:31.601 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:27:31.617 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:27:31.618 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:27:31.618 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:27:31.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:27:31.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:27:31.867 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:27:32.208 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:27:32.208 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:27:32.238 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:27:32.277 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:27:32.346 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:27:32.347 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:27:32.384 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:27:32.415 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:27:32.443 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T19:27:32.443 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/59dde6b4-8a79-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T19:27:32.446 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T19:27:32.487 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:27:33.588 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:29:33.645 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:29:33.700 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:29:34.770 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:29:34.791 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:29:34.791 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:29:34.792 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:29:34.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:29:35.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:29:35.055 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:29:35.378 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:29:35.378 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:29:35.410 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:29:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:29:35.516 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:29:35.517 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:29:35.558 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:29:35.585 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:29:35.614 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T19:29:35.614 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/59dde6b4-8a79-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T19:29:35.615 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T19:29:35.652 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:29:36.760 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:31:36.815 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:31:36.855 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:31:38.971 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:31:38.989 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:31:38.989 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:31:38.989 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:31:38.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:31:39.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:31:39.239 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:31:39.571 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:31:39.571 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:31:39.603 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:31:39.642 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:31:39.713 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:31:39.714 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:31:39.759 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:31:39.789 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:31:39.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T19:31:39.813 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/59dde6b4-8a79-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T19:31:39.815 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T19:31:39.859 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:31:41.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:33:41.941 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:33:41.980 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:33:43.790 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:33:43.806 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:33:43.807 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:33:43.807 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:33:43.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:33:44.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:33:44.059 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:33:44.394 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:33:44.394 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:33:44.429 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:33:44.464 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:33:44.533 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:33:44.534 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:33:44.577 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:33:44.604 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:33:44.633 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T19:33:44.633 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/59dde6b4-8a79-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T19:33:44.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T19:33:44.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:33:45.674 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:35:45.727 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:35:45.767 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:35:46.784 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:35:46.802 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:35:46.802 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:35:46.802 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:35:46.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:35:47.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:35:47.065 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:36:05.570 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T19:36:05.609 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:36:06.627 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:38:06.695 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:38:06.731 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:38:07.684 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:38:07.700 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:38:07.700 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:38:07.700 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:38:07.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:38:07.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:38:07.949 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:38:26.371 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T19:38:26.411 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:38:27.449 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:40:27.504 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:40:27.616 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:40:28.686 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:40:28.703 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:40:28.703 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:40:28.703 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:40:28.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:40:28.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:40:28.955 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:40:29.258 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T19:40:29.290 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:40:29.290 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:40:29.320 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:40:29.320 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:40:29.321 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:40:29.361 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:40:29.377 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T19:40:29.432 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:40:29.432 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:40:29.449 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:40:29.449 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:40:29.451 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:40:29.504 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:40:29.509 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T19:40:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:40:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:40:29.592 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:40:29.592 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:40:29.631 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:40:30.680 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:42:30.730 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:42:30.769 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:42:31.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:42:31.833 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:42:31.833 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:42:31.833 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:42:31.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:42:32.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:42:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:43:32.058 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T19:43:32.114 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:43:33.255 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:45:33.311 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:45:33.353 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:45:34.476 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:45:34.493 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:45:34.493 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:45:34.493 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:45:34.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:45:34.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:45:34.743 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:45:35.041 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T19:45:35.073 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:45:35.073 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:45:35.096 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:45:35.096 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:45:35.097 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:45:35.142 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:45:35.213 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:45:35.213 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:45:35.257 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:45:35.284 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:45:35.312 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T19:45:35.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:45:35.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:45:35.368 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:45:35.368 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:45:35.408 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:45:36.446 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:47:36.505 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:47:36.549 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:47:37.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:47:37.663 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:47:37.663 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:47:37.664 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:47:37.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:47:37.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:47:37.942 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:47:38.251 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:47:38.251 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:47:38.282 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:47:38.320 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:47:38.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:47:38.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:47:38.434 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:47:38.460 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:47:38.529 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:47:38.529 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:47:38.586 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:47:39.674 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:49:39.731 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:49:39.770 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:49:40.784 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:49:40.800 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:49:40.800 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:49:40.800 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:49:40.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:49:41.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:49:41.053 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:49:41.384 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:49:41.384 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:49:41.414 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:49:41.455 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:49:41.524 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:49:41.524 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:49:41.564 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:49:41.592 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:49:41.715 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2021-03-21T19:49:41.742 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2021-03-21T19:49:41.800 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T19:49:41.800 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:49:41.934 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:49:42.953 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:51:43.005 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:51:43.044 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:51:44.110 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:51:44.133 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:51:44.133 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:51:44.134 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:51:44.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:51:44.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:51:44.390 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:51:44.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T19:51:44.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:51:44.769 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:51:44.810 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T19:51:44.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T19:51:44.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T19:51:44.926 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:51:44.959 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T19:51:44.983 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-21T19:51:44.984 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-21T19:51:44.985 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T19:51:45.032 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:51:46.198 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:53:46.251 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:53:46.296 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:53:47.415 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:53:47.435 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:53:47.435 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:53:47.436 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:53:47.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:53:47.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:53:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:53:50.722 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T19:53:50.760 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:53:51.846 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:55:51.899 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:55:51.937 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:55:52.971 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:55:52.987 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:55:52.987 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:55:52.988 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:55:52.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:55:53.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:55:53.236 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:56:08.618 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T19:56:09.716 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T19:56:10.813 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T19:56:11.910 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T19:56:13.007 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T19:56:14.106 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T19:56:15.166 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T19:56:16.262 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T19:56:17.359 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T19:56:18.418 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T19:56:18.418 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T19:56:18.457 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:56:19.532 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:58:19.601 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T19:58:19.638 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:58:20.677 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T19:58:20.694 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T19:58:20.694 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T19:58:20.694 INFO:teuthology.misc:Compressing logs... 2021-03-21T19:58:20.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:58:20.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T19:58:20.949 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T19:58:21.858 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T19:58:21.896 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T19:58:22.915 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:00:22.966 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:00:23.011 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:00:24.153 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:00:24.170 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:00:24.171 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:00:24.171 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:00:24.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:00:24.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:00:24.440 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:00:24.694 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T20:00:24.726 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:00:24.726 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:00:24.751 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:00:24.751 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:00:24.753 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:00:24.796 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:00:24.811 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T20:00:24.864 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:00:24.864 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:00:24.879 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:00:24.879 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:00:24.882 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:00:24.931 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:00:24.938 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T20:00:24.999 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:00:25.000 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:00:25.016 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:00:25.016 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:00:25.056 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:00:26.224 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:02:26.283 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:02:26.326 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:02:27.419 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:02:27.436 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:02:27.436 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:02:27.436 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:02:27.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:02:27.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:02:27.684 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:03:27.657 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T20:03:27.698 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:03:28.806 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:05:28.861 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:05:28.899 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:05:29.983 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:05:30.001 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:05:30.001 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:05:30.001 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:05:30.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:05:30.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:05:30.269 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:05:30.551 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T20:05:30.583 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:05:30.584 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:05:30.611 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:05:30.611 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:05:30.612 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:05:30.654 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:05:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:05:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:05:30.766 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:05:30.802 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:05:30.819 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T20:05:30.871 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:05:30.872 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:05:30.886 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:05:30.886 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:05:30.929 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:05:32.104 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:07:32.162 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:07:32.201 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:07:33.270 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:07:33.287 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:07:33.287 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:07:33.288 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:07:33.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:07:33.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:07:33.541 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:07:33.836 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T20:07:33.869 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:07:33.869 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:07:33.896 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:07:33.896 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:07:33.898 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:07:33.940 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:07:34.013 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:07:34.013 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:07:34.055 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:07:34.083 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:07:34.114 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T20:07:34.153 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:07:34.153 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:07:34.171 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:07:34.171 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:07:34.214 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:07:35.352 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:09:35.408 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:09:35.447 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:09:36.467 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:09:36.483 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:09:36.484 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:09:36.484 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:09:36.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:09:36.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:09:36.734 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:09:37.036 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T20:09:37.068 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:09:37.068 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:09:37.096 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:09:37.097 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:09:37.099 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:09:37.139 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:09:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:09:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:09:37.251 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:09:37.279 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:09:37.353 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:09:37.353 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:09:37.453 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:09:38.462 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:11:38.517 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:11:38.551 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:11:39.654 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:11:39.671 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:11:39.671 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:11:39.671 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:11:39.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:11:39.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:11:39.935 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:11:40.191 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T20:11:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:11:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:11:40.253 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:11:40.253 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:11:40.255 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:11:40.288 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:11:40.358 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:11:40.358 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:11:40.395 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:11:40.426 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:11:40.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:11:40.471 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/e3d58b90-8a80-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:11:40.473 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:11:40.511 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:11:41.506 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:13:41.560 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:13:41.600 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:13:42.723 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:13:42.741 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:13:42.741 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:13:42.741 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:13:42.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:13:42.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:13:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:13:43.299 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T20:13:43.332 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:13:43.332 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:13:43.358 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:13:43.359 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:13:43.361 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:13:43.403 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:13:43.474 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:13:43.474 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:13:43.515 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:13:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:13:43.569 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:13:43.665 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/e3d58b90-8a80-11eb-90a1-001a4aab830c/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T20:13:43.760 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/e3d58b90-8a80-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:13:43.761 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:13:43.805 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:13:44.914 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:15:44.973 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:15:45.010 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:15:46.062 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:15:46.079 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:15:46.079 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:15:46.080 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:15:46.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:15:46.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:15:46.329 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:15:46.634 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T20:15:46.666 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:15:46.666 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:15:46.693 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:15:46.693 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:15:46.696 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:15:46.737 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:15:46.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:15:46.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:15:46.850 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:15:46.877 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:15:46.905 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:15:46.905 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/e3d58b90-8a80-11eb-90a1-001a4aab830c/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T20:15:46.905 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/e3d58b90-8a80-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:15:46.906 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:15:46.991 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:15:47.988 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:17:48.041 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:17:48.111 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:17:49.212 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:17:49.227 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:17:49.228 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:17:49.228 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:17:49.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:17:49.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:17:49.494 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:18:24.070 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T20:18:24.108 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:18:25.122 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:20:25.184 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:20:25.230 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:20:26.289 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:20:26.307 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:20:26.307 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:20:26.307 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:20:26.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:20:26.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:20:26.556 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:20:29.602 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T20:20:29.643 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:20:30.718 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:22:30.770 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:22:30.806 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:22:31.842 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:22:31.863 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:22:31.863 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:22:31.863 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:22:31.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:22:32.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:22:32.113 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:22:32.144 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T20:22:33.202 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T20:22:34.261 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T20:22:35.357 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T20:22:36.454 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T20:22:37.550 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T20:23:38.552 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T20:23:38.623 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:23:39.748 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:25:39.896 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:25:39.934 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:25:40.983 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:25:41.001 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:25:41.001 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:25:41.001 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:25:41.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:25:41.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:25:41.267 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:25:41.553 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T20:25:41.584 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:25:41.584 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:25:41.618 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:25:41.618 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:25:41.619 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:25:41.654 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:25:41.677 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T20:25:41.725 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:25:41.726 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:25:41.744 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:25:41.744 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:25:41.746 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:25:41.794 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:25:41.801 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T20:25:41.862 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:25:41.862 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:25:41.877 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:25:41.878 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:25:41.918 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:25:42.971 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:27:43.028 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:27:43.069 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:27:44.139 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:27:44.156 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:27:44.156 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:27:44.157 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:27:44.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:27:44.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:27:44.416 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:28:44.389 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T20:28:44.432 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:28:45.527 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:30:45.595 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:30:45.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:30:46.709 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:30:46.726 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:30:46.727 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:30:46.727 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:30:46.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:30:46.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:30:46.993 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:30:47.342 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T20:30:47.379 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:30:47.380 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:30:47.394 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:30:47.394 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:30:47.395 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:30:47.451 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:30:47.521 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:30:47.521 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:30:47.560 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:30:47.594 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:30:47.600 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T20:30:47.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:30:47.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:30:47.685 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:30:47.685 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:30:47.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:30:48.780 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:32:48.836 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:32:48.874 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:32:49.903 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:32:49.937 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:32:49.937 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:32:49.938 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:32:49.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:32:50.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:32:50.198 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:32:50.527 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:32:50.527 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:32:50.559 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:32:50.599 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:32:50.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:32:50.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:32:50.707 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:32:50.738 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:32:50.806 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:32:50.807 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:32:50.865 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:32:51.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:34:51.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:34:52.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:34:53.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:34:53.223 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:34:53.223 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:34:53.224 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:34:53.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:34:53.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:34:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:34:53.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:34:53.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:34:53.857 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:34:53.899 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:34:53.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:34:53.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:34:54.008 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:34:54.044 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:34:54.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:34:54.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:34:54.168 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:34:55.234 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:36:55.288 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:36:55.325 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:36:56.317 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:36:56.333 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:36:56.333 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:36:56.333 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:36:56.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:36:56.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:36:56.595 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:36:56.898 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:36:56.899 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:36:56.926 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:36:56.967 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:36:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:36:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:36:57.077 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:36:57.102 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:36:57.170 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T20:36:57.171 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:36:57.239 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:36:58.203 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:38:58.255 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:38:58.293 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:38:59.302 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:38:59.318 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:38:59.318 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:38:59.318 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:38:59.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:38:59.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:38:59.567 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:38:59.890 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:38:59.890 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:38:59.919 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:38:59.961 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:39:00.046 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:39:00.046 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:39:00.082 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:39:00.115 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:39:00.140 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:39:00.178 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:39:00.215 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:39:01.252 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:41:01.306 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:41:01.345 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:41:02.409 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:41:02.441 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:41:02.441 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:41:02.442 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:41:02.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:41:02.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:41:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:41:03.026 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:41:03.026 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:41:03.060 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:41:03.097 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:41:03.169 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:41:03.169 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:41:03.201 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:41:03.236 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:41:03.257 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:41:03.258 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:41:03.259 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:41:03.298 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:41:04.338 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:43:04.390 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:43:04.428 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:43:05.441 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:43:05.458 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:43:05.458 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:43:05.458 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:43:05.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:43:05.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:43:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:43:06.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:43:06.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:43:06.062 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:43:06.104 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:43:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:43:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:43:06.212 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:43:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:43:06.269 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:43:06.286 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.11.log: file size changed while zipping 2021-03-21T20:43:06.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:43:06.357 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:43:06.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:43:07.410 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:45:07.478 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:45:07.517 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:45:08.524 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:45:08.540 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:45:08.540 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:45:08.540 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:45:08.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:45:08.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:45:08.790 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:45:09.127 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:45:09.127 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:45:09.158 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:45:09.198 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:45:09.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:45:09.355 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:45:09.398 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:45:09.436 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:45:09.436 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:45:09.437 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:45:09.439 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:45:09.478 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:45:10.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:47:10.577 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:47:10.616 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:47:11.646 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:47:11.663 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:47:11.664 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:47:11.664 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:47:11.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:47:11.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:47:11.916 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:47:12.251 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:47:12.251 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:47:12.287 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:47:12.321 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:47:12.391 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:47:12.391 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:47:12.431 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:47:12.460 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:47:12.487 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:47:12.487 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T20:47:12.487 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T20:47:12.487 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T20:47:12.487 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:47:12.487 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T20:47:12.488 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:47:12.525 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:47:13.562 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:49:13.616 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:49:13.652 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:49:14.638 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:49:14.656 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:49:14.656 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:49:14.656 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:49:14.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:49:14.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:49:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:49:15.237 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:49:15.237 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:49:15.268 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:49:15.306 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:49:15.375 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:49:15.376 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:49:15.420 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:49:15.447 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:49:15.477 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:49:15.477 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T20:49:15.477 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T20:49:15.477 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T20:49:15.477 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:49:15.477 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T20:49:15.480 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:49:15.516 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:49:16.505 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:51:16.558 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:51:16.596 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:51:17.621 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:51:17.639 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:51:17.640 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:51:17.640 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:51:17.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:51:17.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:51:17.890 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:51:18.219 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:51:18.219 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:51:18.250 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:51:18.290 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:51:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:51:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:51:18.403 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:51:18.430 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:51:18.458 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:51:18.458 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T20:51:18.458 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T20:51:18.458 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T20:51:18.458 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:51:18.458 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T20:51:18.462 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:51:18.498 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:51:19.488 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:53:19.541 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:53:19.638 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:53:20.707 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:53:20.727 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:53:20.875 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:53:20.875 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:53:20.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:53:21.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:53:21.126 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:53:21.463 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:53:21.463 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:53:21.495 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:53:21.533 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:53:21.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:53:21.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:53:21.646 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:53:21.673 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:53:21.702 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:53:21.703 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T20:53:21.703 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T20:53:21.703 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T20:53:21.703 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:53:21.703 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T20:53:21.703 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:53:21.745 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:53:22.808 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:55:22.866 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:55:22.907 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:55:24.012 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:55:24.029 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:55:24.029 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:55:24.030 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:55:24.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:55:24.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:55:24.299 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:55:24.617 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:55:24.617 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:55:24.654 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:55:24.686 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:55:24.754 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:55:24.755 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:55:24.793 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:55:24.821 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:55:24.850 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:55:24.850 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T20:55:24.850 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T20:55:24.851 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T20:55:24.851 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:55:24.851 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T20:55:24.853 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:55:24.891 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:55:26.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:57:26.065 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:57:26.104 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:57:27.110 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:57:27.126 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:57:27.126 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:57:27.126 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:57:27.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:57:27.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:57:27.374 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:57:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:57:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:57:27.738 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:57:27.777 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:57:27.848 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:57:27.848 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:57:27.891 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:57:27.919 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:57:27.947 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:57:27.948 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T20:57:27.948 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T20:57:27.948 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T20:57:27.948 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:57:27.948 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T20:57:27.949 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:57:27.986 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:57:29.017 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:59:29.076 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T20:59:29.112 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:59:30.122 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T20:59:30.137 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T20:59:30.138 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T20:59:30.138 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:59:30.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:59:30.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T20:59:30.394 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T20:59:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T20:59:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:59:30.762 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:59:30.799 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T20:59:30.869 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T20:59:30.869 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T20:59:30.909 INFO:teuthology.misc:Compressing logs... 2021-03-21T20:59:30.938 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T20:59:30.964 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T20:59:30.964 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T20:59:30.964 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T20:59:30.964 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T20:59:30.964 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T20:59:30.964 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T20:59:30.965 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T20:59:31.002 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T20:59:31.916 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:01:31.969 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:01:32.007 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:01:32.991 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:01:33.007 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:01:33.007 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:01:33.007 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:01:33.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:01:33.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:01:33.276 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:01:33.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:01:33.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:01:33.618 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:01:33.657 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:01:33.727 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:01:33.727 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:01:33.765 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:01:33.795 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:01:33.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T21:01:33.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T21:01:33.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T21:01:33.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T21:01:33.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T21:01:33.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T21:01:33.824 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:01:33.862 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:01:34.865 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:03:34.918 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:03:34.955 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:03:35.983 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:03:36.000 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:03:36.000 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:03:36.000 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:03:36.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:03:36.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:03:36.248 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:03:36.570 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:03:36.570 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:03:36.599 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:03:36.640 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:03:36.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:03:36.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:03:36.754 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:03:36.781 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:03:36.811 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T21:03:36.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T21:03:36.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T21:03:36.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T21:03:36.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T21:03:36.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T21:03:36.812 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:03:36.850 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:03:37.859 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:05:37.915 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:05:37.954 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:05:39.007 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:05:39.024 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:05:39.024 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:05:39.024 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:05:39.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:05:39.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:05:39.274 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:05:39.614 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:05:39.615 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:05:39.645 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:05:39.684 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:05:39.754 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:05:39.755 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:05:39.796 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:05:39.823 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:05:39.852 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T21:05:39.852 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T21:05:39.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T21:05:39.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T21:05:39.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T21:05:39.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/c4da744a-8a84-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T21:05:39.854 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:05:39.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:05:40.927 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:07:40.984 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:07:41.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:07:42.021 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:07:42.037 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:07:42.037 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:07:42.037 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:07:42.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:07:42.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:07:42.299 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:08:00.578 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T21:08:00.619 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:08:01.664 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:10:01.727 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:10:01.762 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:10:02.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:10:02.792 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:10:02.792 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:10:02.792 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:10:02.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:10:03.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:10:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:10:04.742 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T21:10:04.778 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:10:05.753 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:12:05.811 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:12:05.849 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:12:06.891 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:12:06.907 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:12:06.907 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:12:06.907 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:12:06.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:12:07.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:12:07.156 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:12:07.362 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:12:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:12:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:12:07.454 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:12:07.455 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:12:07.456 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:12:07.483 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:12:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:12:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:12:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:12:07.583 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:12:07.583 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:12:07.584 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:12:07.630 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:12:07.647 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:12:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T21:12:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:12:07.724 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T21:12:07.724 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:12:07.761 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:12:08.782 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:14:08.839 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:14:08.881 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:14:09.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:14:09.951 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:14:09.951 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:14:09.952 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:14:09.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:14:10.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:14:10.215 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:14:10.398 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:14:10.428 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:14:10.428 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:14:10.463 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:14:10.463 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:14:10.465 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:14:10.498 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:14:10.570 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:14:10.570 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:14:10.632 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:14:10.660 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:14:10.669 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:14:10.729 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T21:14:10.729 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:14:10.753 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T21:14:10.753 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:14:10.794 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:14:11.939 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:16:11.992 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:16:12.030 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:16:13.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:16:13.146 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:16:13.146 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:16:13.146 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:16:13.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:16:13.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:16:13.403 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:16:13.610 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:16:13.641 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:16:13.641 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:16:13.680 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:16:13.680 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:16:13.682 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:16:13.717 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:16:13.748 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T21:16:13.750 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:16:13.805 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:16:14.833 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:18:14.890 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:18:14.928 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:18:16.960 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:18:16.979 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:18:16.979 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:18:16.980 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:18:16.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:18:17.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:18:17.235 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:18:17.435 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:18:17.467 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:18:17.468 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:18:17.503 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:18:17.503 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:18:17.505 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:18:17.541 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:18:17.569 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T21:18:17.571 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:18:17.615 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:18:19.872 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:20:19.928 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:20:19.968 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:20:22.008 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:20:22.027 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:20:22.027 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:20:22.027 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:20:22.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:20:22.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:20:22.300 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:20:22.519 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:20:22.519 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:20:22.557 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:20:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:20:22.622 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T21:20:22.623 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:20:22.702 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:20:24.392 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:22:24.445 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:22:24.492 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:22:26.327 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:22:26.344 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:22:26.345 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:22:26.345 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:22:26.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:22:26.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:22:26.595 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:22:26.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:22:26.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:22:26.891 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:22:26.918 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:22:26.959 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T21:22:26.960 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:22:26.999 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:22:28.246 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:24:28.301 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:24:28.345 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:24:30.110 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:24:30.127 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:24:30.127 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:24:30.128 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:24:30.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:24:30.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:24:30.384 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:25:30.358 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T21:25:30.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:25:31.412 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:27:31.470 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:27:31.508 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:27:32.602 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:27:32.619 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:27:32.619 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:27:32.619 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:27:32.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:27:32.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:27:32.883 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:27:35.906 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T21:27:35.944 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:27:37.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:29:37.096 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:29:37.188 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:29:38.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:29:38.300 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:29:38.300 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:29:38.300 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:29:38.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:29:38.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:29:38.547 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:29:41.570 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T21:29:41.608 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:29:42.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:31:42.759 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:31:42.800 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:31:43.851 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:31:43.868 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:31:43.869 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:31:43.869 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:31:43.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:31:44.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:31:44.117 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:31:44.420 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T21:31:44.451 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:31:44.452 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:31:44.481 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:31:44.481 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:31:44.482 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:31:44.521 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:31:44.539 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T21:31:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:31:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:31:44.606 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:31:44.606 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:31:44.609 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:31:44.658 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:31:44.663 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T21:31:44.726 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T21:31:44.726 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:31:44.743 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T21:31:44.743 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:31:44.786 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:31:45.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:33:45.902 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:33:45.944 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:33:47.002 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:33:47.019 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:33:47.019 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:33:47.020 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:33:47.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:33:47.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:33:47.285 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:34:05.763 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T21:34:05.805 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:34:06.963 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:36:07.020 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:36:07.063 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:36:08.145 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:36:08.165 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:36:08.165 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:36:08.165 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:36:08.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:36:08.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:36:08.429 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:36:08.728 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T21:36:08.760 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:36:08.760 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:36:08.785 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:36:08.785 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:36:08.788 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:36:08.831 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:36:08.902 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:36:08.902 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:36:08.938 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:36:08.972 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:36:08.976 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T21:36:09.044 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T21:36:09.044 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:36:09.060 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T21:36:09.060 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:36:09.100 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:36:10.174 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:38:10.234 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:38:10.275 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:38:11.285 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:38:11.304 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:38:11.304 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:38:11.304 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:38:11.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:38:11.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:38:11.555 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:38:11.842 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T21:38:11.872 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:38:11.872 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:38:11.897 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:38:11.898 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:38:11.900 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:38:11.943 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:38:12.014 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:38:12.015 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:38:12.054 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:38:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:38:12.106 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T21:38:12.152 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T21:38:12.152 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:38:12.167 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T21:38:12.167 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:38:12.206 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:38:13.242 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:40:13.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:40:13.337 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:40:14.418 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:40:14.435 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:40:14.436 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:40:14.436 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:40:14.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:40:14.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:40:14.703 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:40:15.033 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:40:15.034 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:40:15.066 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:40:15.105 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:40:15.174 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:40:15.174 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:40:15.225 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:40:15.251 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:40:15.337 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T21:40:15.338 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:40:15.449 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:40:16.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:42:16.583 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:42:16.652 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:42:17.741 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:42:17.761 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:42:17.761 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:42:17.761 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:42:17.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:42:17.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:42:18.012 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:42:18.377 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:42:18.378 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:42:18.401 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:42:18.449 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:42:18.518 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:42:18.518 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:42:18.554 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:42:18.587 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:42:18.605 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T21:42:18.606 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T21:42:18.606 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T21:42:18.607 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:42:18.649 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:42:19.709 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:44:19.769 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:44:19.812 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:44:20.920 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:44:20.937 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:44:20.937 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:44:20.937 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:44:20.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:44:21.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:44:21.192 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:44:21.615 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:44:21.616 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:44:21.644 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:44:21.686 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:44:21.758 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:44:21.758 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:44:21.800 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:44:21.829 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:44:21.861 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T21:44:21.861 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T21:44:21.862 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T21:44:21.863 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:44:21.903 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:44:22.980 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:46:23.036 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:46:23.075 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:46:24.147 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:46:24.165 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:46:24.165 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:46:24.165 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:46:24.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:46:24.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:46:24.431 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:46:25.094 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:46:25.094 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:46:25.126 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:46:25.162 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:46:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:46:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:46:25.282 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:46:25.316 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:46:25.343 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T21:46:25.343 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T21:46:25.343 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T21:46:25.344 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:46:25.382 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:46:26.420 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:48:26.474 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:48:26.513 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:48:27.532 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:48:27.553 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:48:27.553 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:48:27.554 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:48:27.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:48:27.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:48:27.804 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:48:28.368 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:48:28.368 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:48:28.455 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:48:28.484 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:48:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:48:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:48:28.592 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:48:28.623 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:48:28.644 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T21:48:28.644 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T21:48:28.644 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T21:48:28.645 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:48:28.685 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:48:29.721 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:50:29.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:50:29.816 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:50:30.876 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:50:30.894 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:50:30.895 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:50:30.895 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:50:30.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:50:31.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:50:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:50:31.660 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:50:31.660 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:50:31.688 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:50:31.732 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:50:31.802 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:50:31.802 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:50:31.840 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:50:31.872 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:50:32.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T21:50:35.237 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T21:50:37.605 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T21:50:38.205 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:50:38.245 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:50:39.350 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:52:39.407 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:52:39.448 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:52:40.513 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:52:40.544 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:52:40.544 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:52:40.544 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:52:40.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:52:40.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:52:40.822 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:52:41.767 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:52:41.767 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:52:41.883 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:52:41.911 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:52:41.982 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:52:41.982 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:52:42.021 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:52:42.052 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:52:42.090 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T21:52:42.090 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T21:52:42.091 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T21:52:42.092 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:52:42.133 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:52:43.217 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:54:43.271 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:54:43.309 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:54:44.371 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:54:44.390 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:54:44.391 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:54:44.391 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:54:44.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:54:44.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:54:44.645 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:54:45.036 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:54:45.036 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:54:45.057 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:54:45.105 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:54:45.175 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:54:45.175 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:54:45.215 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:54:45.250 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:54:45.271 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T21:54:45.272 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T21:54:45.272 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T21:54:45.273 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:54:45.312 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:54:46.323 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:56:46.378 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:56:46.425 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:56:47.550 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:56:47.568 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:56:47.568 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:56:47.568 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:56:47.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:56:47.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:56:47.820 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:56:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:56:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:56:48.227 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:56:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:56:48.338 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:56:48.338 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:56:48.375 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:56:48.409 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:56:48.429 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T21:56:48.429 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T21:56:48.430 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T21:56:48.431 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:56:48.475 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:56:49.567 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:58:49.625 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T21:58:49.664 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:58:50.686 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T21:58:50.706 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T21:58:50.706 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T21:58:50.706 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:58:50.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:58:50.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T21:58:50.975 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T21:58:51.300 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T21:58:51.300 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:58:51.332 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:58:51.369 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T21:58:51.439 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T21:58:51.439 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T21:58:51.473 INFO:teuthology.misc:Compressing logs... 2021-03-21T21:58:51.506 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T21:58:51.534 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T21:58:51.534 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T21:58:51.535 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T21:58:51.536 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T21:58:51.572 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T21:58:52.579 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:00:52.633 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:00:52.673 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:00:53.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:00:53.738 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:00:53.739 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:00:53.739 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:00:53.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:00:53.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:00:53.988 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:00:54.913 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:00:54.914 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:00:55.046 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:00:55.076 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:00:55.145 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:00:55.146 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:00:55.183 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:00:55.213 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:00:55.235 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:00:55.236 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:00:55.236 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:00:55.237 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:00:55.275 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:00:56.298 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:02:56.368 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:02:56.414 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:02:57.531 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:02:57.551 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:02:57.552 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:02:57.552 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:02:57.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:02:57.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:02:57.807 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:02:58.165 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:02:58.165 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:02:58.191 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:02:58.236 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:02:58.305 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:02:58.306 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:02:58.338 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:02:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:02:58.390 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:02:58.391 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:02:58.391 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:02:58.392 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:02:58.431 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:02:59.492 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:04:59.545 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:04:59.635 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:05:00.704 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:05:00.721 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:05:00.721 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:05:00.721 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:05:00.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:05:00.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:05:00.988 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:05:01.371 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:05:01.371 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:05:01.396 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:05:01.455 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:05:01.522 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:05:01.523 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:05:01.567 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:05:01.606 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:05:01.618 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:05:01.618 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:05:01.618 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:05:01.619 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:05:01.658 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:05:02.804 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:07:02.856 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:07:02.895 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:07:03.934 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:07:03.951 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:07:03.952 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:07:03.952 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:07:03.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:07:04.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:07:04.202 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:07:04.632 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:07:04.633 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:07:04.680 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:07:04.708 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:07:04.777 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:07:04.778 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:07:04.810 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:07:04.847 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:07:04.864 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:07:04.864 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:07:04.864 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:07:04.865 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:07:04.906 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:07:05.932 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:09:05.996 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:09:06.034 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:09:07.094 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:09:07.111 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:09:07.111 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:09:07.111 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:09:07.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:09:07.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:09:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:09:07.735 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:09:07.735 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:09:07.761 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:09:07.805 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:09:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:09:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:09:07.911 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:09:07.942 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:09:07.963 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:09:07.963 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:09:07.963 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:09:07.964 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:09:08.001 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:09:09.050 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:11:09.106 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:11:09.145 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:11:10.234 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:11:10.251 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:11:10.252 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:11:10.252 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:11:10.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:11:10.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:11:10.522 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:11:10.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:11:10.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:11:10.923 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:11:10.954 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:11:11.025 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:11:11.025 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:11:11.074 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:11:11.109 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:11:11.128 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:11:11.128 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:11:11.128 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:11:11.129 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:11:11.167 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:11:12.244 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:13:12.297 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:13:12.336 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:13:13.385 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:13:13.402 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:13:13.402 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:13:13.402 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:13:13.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:13:13.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:13:13.658 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:13:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:13:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:13:14.035 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:13:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:13:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:13:14.143 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:13:14.183 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:13:14.211 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:13:14.241 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:13:14.241 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:13:14.242 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:13:14.243 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:13:14.282 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:13:15.345 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:15:15.400 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:15:15.440 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:15:16.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:15:16.516 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:15:16.517 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:15:16.517 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:15:16.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:15:16.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:15:16.771 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:15:17.105 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:15:17.106 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:15:17.139 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:15:17.177 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:15:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:15:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:15:17.284 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:15:17.314 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:15:17.341 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:15:17.341 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:15:17.341 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:15:17.343 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:15:17.380 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:15:18.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:17:18.422 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:17:18.458 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:17:19.477 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:17:19.494 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:17:19.494 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:17:19.495 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:17:19.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:17:19.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:17:19.758 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:17:20.068 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:17:20.069 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:17:20.100 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:17:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:17:20.208 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:17:20.208 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:17:20.249 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:17:20.275 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:17:20.306 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:17:20.306 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:17:20.306 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:17:20.308 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:17:20.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:17:21.373 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:19:21.427 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:19:21.466 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:19:22.593 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:19:22.609 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:19:22.610 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:19:22.610 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:19:22.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:19:22.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:19:22.862 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:19:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:19:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:19:23.243 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:19:23.281 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:19:23.352 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:19:23.352 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:19:23.392 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:19:23.422 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:19:23.450 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:19:23.450 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:19:23.450 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:19:23.452 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:19:23.491 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:19:24.595 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:21:24.653 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:21:24.696 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:21:25.738 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:21:25.756 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:21:25.756 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:21:25.756 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:21:25.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:21:25.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:21:26.010 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:21:26.357 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:21:26.358 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:21:26.390 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:21:26.427 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:21:26.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:21:26.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:21:26.538 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:21:26.564 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:21:26.593 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:21:26.593 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:21:26.593 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:21:26.595 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:21:26.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:21:27.706 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:23:27.768 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:23:27.809 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:23:28.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:23:28.869 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:23:28.869 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:23:28.869 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:23:28.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:23:29.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:23:29.139 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:23:29.470 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:23:29.471 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:23:29.502 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:23:29.541 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:23:29.610 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:23:29.611 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:23:29.651 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:23:29.679 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:23:29.704 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:23:29.704 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:23:29.705 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:23:29.706 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:23:29.744 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:23:30.779 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:25:30.848 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:25:30.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:25:32.749 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:25:32.766 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:25:32.766 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:25:32.767 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:25:32.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:25:32.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:25:33.017 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:25:33.362 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:25:33.362 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:25:33.392 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:25:33.433 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:25:33.503 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:25:33.503 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:25:33.546 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:25:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:25:33.606 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:25:33.606 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:25:33.606 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:25:33.608 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:25:33.648 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:25:35.223 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:27:35.285 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:27:35.323 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:27:37.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:27:37.214 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:27:37.214 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:27:37.215 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:27:37.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:27:37.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:27:37.467 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:27:37.803 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:27:37.803 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:27:37.837 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:27:37.873 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:27:37.943 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:27:37.943 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:27:37.987 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:27:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:27:38.030 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:27:38.030 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:27:38.030 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:27:38.033 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:27:38.073 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:27:39.182 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:29:39.238 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:29:39.276 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:29:40.317 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:29:40.333 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:29:40.334 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:29:40.334 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:29:40.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:29:40.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:29:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:29:40.942 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:29:40.942 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:29:40.973 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:29:41.012 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:29:41.081 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:29:41.081 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:29:41.121 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:29:41.149 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:29:41.176 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:29:41.177 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:29:41.177 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:29:41.179 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:29:41.216 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:29:42.292 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:31:42.350 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:31:42.390 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:31:43.430 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:31:43.449 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:31:43.449 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:31:43.449 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:31:43.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:31:43.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:31:43.702 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:31:44.041 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:31:44.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:31:44.073 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:31:44.112 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:31:44.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:31:44.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:31:44.226 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:31:44.256 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:31:44.280 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:31:44.280 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:31:44.280 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:31:44.282 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:31:44.321 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:31:45.395 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:33:45.486 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:33:45.524 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:33:46.583 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:33:46.601 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:33:46.602 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:33:46.602 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:33:46.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:33:46.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:33:46.858 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:33:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:33:47.195 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:33:47.224 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:33:47.265 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:33:47.335 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:33:47.335 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:33:47.376 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:33:47.404 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:33:47.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-21T22:33:47.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-21T22:33:47.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-21T22:33:47.434 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:33:47.475 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:33:48.509 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:35:48.568 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:35:48.607 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:35:49.601 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:35:49.618 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:35:49.619 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:35:49.619 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:35:49.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:35:49.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:35:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:35:52.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T22:35:52.937 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:35:53.978 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:37:54.031 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:37:54.073 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:37:55.096 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:37:55.112 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:37:55.113 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:37:55.113 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:37:55.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:37:55.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:37:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:37:55.393 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T22:37:56.491 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T22:37:57.590 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T22:37:58.687 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T22:37:59.783 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T22:38:00.884 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T22:38:01.986 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T22:38:03.090 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T22:38:04.201 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T22:38:05.298 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T22:38:05.299 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T22:38:05.341 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:38:06.420 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:40:06.475 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:40:06.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:40:07.616 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:40:07.633 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:40:07.633 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:40:07.633 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:40:07.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:40:07.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:40:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:40:08.170 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:40:08.200 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:40:08.200 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:40:08.249 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:40:08.249 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:40:08.250 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:40:08.281 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:40:08.283 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:40:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:40:08.353 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:40:08.381 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:40:08.381 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:40:08.382 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:40:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:40:08.446 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:40:08.490 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T22:40:08.490 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:40:08.514 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T22:40:08.514 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:40:08.554 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:40:09.647 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:42:09.700 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:42:09.734 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:42:10.695 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:42:10.713 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:42:10.713 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:42:10.713 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:42:10.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:42:10.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:42:10.980 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:42:11.168 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:42:11.197 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:42:11.197 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:42:11.234 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:42:11.234 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:42:11.235 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:42:11.264 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:42:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T22:42:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:42:11.393 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:42:11.418 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:42:11.436 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T22:42:11.485 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T22:42:11.485 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:42:11.509 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T22:42:11.509 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:42:11.546 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:42:12.498 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:44:12.553 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:44:12.596 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:44:13.664 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:44:13.684 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:44:13.684 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:44:13.684 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:44:13.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:44:13.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:44:13.930 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:44:14.127 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:44:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:44:14.158 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:44:14.193 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:44:14.194 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:44:14.194 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:44:14.230 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:44:14.258 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T22:44:14.260 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:44:14.301 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:44:15.292 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:46:15.347 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:46:15.434 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:46:16.456 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:46:16.475 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:46:16.475 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:46:16.475 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:46:16.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:46:16.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:46:16.725 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:46:16.912 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:46:16.943 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:46:16.943 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:46:16.976 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:46:16.977 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:46:16.978 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:46:17.012 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:46:17.043 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T22:46:17.045 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:46:17.103 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:46:18.120 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:48:18.171 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:48:18.231 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:48:19.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:48:19.286 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:48:19.286 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:48:19.286 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:48:19.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:48:19.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:48:19.551 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:48:19.775 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:48:19.775 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:48:19.814 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:48:19.844 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:48:19.880 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T22:48:19.881 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:48:19.917 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:48:20.962 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:50:21.016 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:50:21.052 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:50:22.102 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:50:22.122 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:50:22.122 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:50:22.122 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:50:22.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:50:22.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:50:22.375 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:50:22.615 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:50:22.616 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:50:22.654 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:50:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:50:22.720 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T22:50:22.721 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:50:22.761 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:50:23.777 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:52:23.835 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:52:23.874 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:52:24.948 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:52:24.965 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:52:24.965 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:52:24.965 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:52:24.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:52:25.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:52:25.217 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:52:25.442 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:52:25.446 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:52:25.479 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:52:25.516 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:52:25.545 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T22:52:25.546 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:52:25.585 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:52:26.669 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:54:26.727 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:54:26.778 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:54:27.910 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:54:27.928 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:54:27.928 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:54:27.928 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:54:27.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:54:28.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:54:28.201 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:54:28.419 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:54:28.422 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:54:28.458 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:54:28.491 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:54:28.523 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T22:54:28.524 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:54:28.565 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:54:29.652 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:56:29.762 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:56:29.802 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:56:30.953 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:56:30.973 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:56:30.973 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:56:30.973 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:56:30.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:56:31.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:56:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:56:31.446 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:56:31.447 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:56:31.485 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:56:31.518 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:56:31.550 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T22:56:31.552 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:56:31.590 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:56:32.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:58:32.698 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T22:58:32.736 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:58:34.085 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T22:58:34.103 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T22:58:34.103 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T22:58:34.103 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:58:34.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:58:34.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T22:58:34.357 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T22:58:34.581 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T22:58:34.581 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T22:58:34.620 INFO:teuthology.misc:Compressing logs... 2021-03-21T22:58:34.651 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T22:58:34.686 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T22:58:34.687 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T22:58:34.731 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T22:58:35.848 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:00:35.906 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:00:35.951 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:00:37.113 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:00:37.130 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:00:37.130 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:00:37.130 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:00:37.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:00:37.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:00:37.403 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:00:37.628 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:00:37.629 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:00:37.668 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:00:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:00:37.733 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T23:00:37.735 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T23:00:37.787 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:00:38.980 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:02:39.039 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:02:39.108 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:02:40.347 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:02:40.371 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:02:40.371 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:02:40.371 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:02:40.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:02:40.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:02:40.629 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:02:40.858 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:02:41.053 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:02:41.092 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:02:41.136 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:02:41.158 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-21T23:02:41.159 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T23:02:41.199 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:02:42.411 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:04:42.465 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:04:42.505 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:04:43.603 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:04:43.622 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:04:43.622 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:04:43.622 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:04:43.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:04:43.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:04:43.874 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:05:02.402 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T23:05:02.442 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:05:03.542 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:07:03.600 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:07:03.639 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:07:04.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:07:04.736 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:07:04.737 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:07:04.737 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:07:04.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:07:04.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:07:05.004 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:07:05.018 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T23:07:06.117 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T23:07:07.215 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T23:07:08.313 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T23:07:09.409 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T23:07:10.511 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T23:07:11.606 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T23:07:12.706 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T23:07:13.807 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T23:07:14.904 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T23:07:14.904 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T23:07:14.944 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:07:15.981 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:09:16.037 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:09:16.083 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:09:18.022 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:09:18.040 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:09:18.040 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:09:18.040 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:09:18.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:09:18.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:09:18.300 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:09:18.694 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T23:09:18.734 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:09:20.332 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:11:20.396 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:11:20.435 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:11:22.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:11:22.210 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:11:22.210 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:11:22.210 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:11:22.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:11:22.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:11:22.460 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:11:22.765 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T23:11:22.798 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:11:22.799 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:11:22.825 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:11:22.825 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:11:22.828 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:11:22.870 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:11:22.884 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T23:11:22.941 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:11:22.941 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:11:22.955 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:11:22.955 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:11:22.957 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:11:23.010 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:11:23.016 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T23:11:23.079 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:11:23.079 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:11:23.095 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:11:23.096 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:11:23.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:11:24.214 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:13:24.270 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:13:24.309 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:13:25.404 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:13:25.421 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:13:25.421 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:13:25.421 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:13:25.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:13:25.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:13:25.692 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:14:00.259 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T23:14:00.299 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:14:01.413 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:16:01.492 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:16:01.533 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:16:02.661 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:16:02.683 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:16:02.683 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:16:02.684 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:16:02.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:16:02.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:16:02.944 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:16:03.260 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T23:16:03.293 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:16:03.294 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:16:03.323 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:16:03.324 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:16:03.326 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:16:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:16:03.436 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:16:03.436 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:16:03.471 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:16:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:16:03.520 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T23:16:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:16:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:16:03.591 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:16:03.591 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:16:03.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:16:04.694 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:18:04.769 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:18:04.810 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:18:06.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:18:06.618 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:18:06.618 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:18:06.618 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:18:06.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:18:06.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:18:06.873 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:18:07.176 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T23:18:07.209 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:18:07.209 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:18:07.237 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:18:07.238 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:18:07.241 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:18:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:18:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:18:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:18:07.388 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:18:07.419 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:18:07.442 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T23:18:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:18:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:18:07.506 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:18:07.506 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:18:07.555 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:18:08.645 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:20:08.707 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:20:08.746 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:20:10.632 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:20:10.649 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:20:10.649 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:20:10.649 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:20:10.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:20:10.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:20:10.916 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:20:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:20:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:20:11.261 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:20:11.299 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:20:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:20:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:20:11.416 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:20:11.442 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:20:11.511 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:20:11.511 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:20:11.574 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:20:12.696 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:22:12.753 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:22:12.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:22:13.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:22:13.949 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:22:13.949 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:22:13.949 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:22:13.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:22:14.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:22:14.202 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:22:14.559 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:22:14.559 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:22:14.583 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:22:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:22:14.701 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:22:14.701 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:22:14.736 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:22:14.770 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:22:14.976 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.11.log: file size changed while zipping 2021-03-21T23:22:15.270 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2021-03-21T23:22:15.303 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:22:15.303 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:22:15.504 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:22:16.558 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:24:16.619 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:24:16.660 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:24:17.732 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:24:17.751 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:24:17.752 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:24:17.752 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:24:17.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:24:17.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:24:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:24:52.547 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T23:24:52.587 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:24:53.651 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:26:53.705 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:26:53.743 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:26:54.756 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:26:54.773 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:26:54.773 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:26:54.774 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:26:54.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:26:54.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:26:55.040 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:26:58.050 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T23:26:58.089 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:26:59.102 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:28:59.155 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:28:59.193 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:29:00.265 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:29:00.283 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:29:00.283 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:29:00.283 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:29:00.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:29:00.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:29:00.537 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:29:00.611 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-21T23:30:01.618 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T23:30:01.658 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:30:02.668 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:32:02.724 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:32:02.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:32:03.843 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:32:03.860 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:32:03.861 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:32:03.861 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:32:03.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:32:04.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:32:04.127 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:32:04.414 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T23:32:04.446 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:32:04.447 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:32:04.482 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:32:04.483 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:32:04.485 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:32:04.534 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:32:04.559 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-21T23:32:04.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:32:04.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:32:04.638 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:32:04.638 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:32:04.640 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:32:04.692 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:32:04.698 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T23:32:04.762 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:32:04.762 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:32:04.779 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:32:04.779 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:32:04.819 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:32:05.894 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:34:05.954 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:34:05.993 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:34:07.031 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:34:07.048 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:34:07.048 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:34:07.049 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:34:07.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:34:07.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:34:07.278 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T23:34:07.316 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:34:08.327 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:36:08.382 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:36:08.420 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:36:09.487 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:36:09.504 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:36:09.504 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:36:09.504 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:36:09.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:36:09.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:36:09.759 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:36:10.073 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T23:36:10.106 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:36:10.106 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:36:10.127 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:36:10.127 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:36:10.129 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:36:10.177 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:36:10.247 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:36:10.248 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:36:10.283 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:36:10.318 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:36:10.327 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T23:36:10.387 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:36:10.387 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:36:10.401 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:36:10.402 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:36:10.442 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:36:11.498 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:38:11.555 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:38:11.594 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:38:12.616 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:38:12.634 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:38:12.634 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:38:12.634 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:38:12.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:38:12.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:38:12.901 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:38:13.194 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-21T23:38:13.228 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:38:13.228 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:38:13.258 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:38:13.259 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:38:13.261 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:38:13.298 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:38:13.368 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:38:13.368 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:38:13.404 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:38:13.437 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:38:13.441 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-21T23:38:13.505 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:38:13.506 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:38:13.522 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:38:13.522 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:38:13.562 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:38:14.681 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:40:14.742 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:40:14.783 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:40:15.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:40:15.872 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:40:15.872 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:40:15.872 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:40:15.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:40:16.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:40:16.134 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:40:16.463 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:40:16.463 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:40:16.499 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:40:16.534 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:40:16.604 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:40:16.604 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:40:16.642 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:40:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:40:16.741 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:40:16.741 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:40:16.797 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:40:17.923 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:42:17.978 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:42:18.018 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:42:19.087 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:42:19.106 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:42:19.106 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:42:19.106 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:42:19.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:42:19.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:42:19.357 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:42:19.683 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:42:19.684 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:42:19.712 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:42:19.755 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:42:19.827 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:42:19.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:42:19.868 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:42:19.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:42:19.965 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:42:19.966 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:42:20.024 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:42:21.045 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:44:21.100 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:44:21.139 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:44:22.299 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:44:22.319 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:44:22.319 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:44:22.319 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:44:22.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:44:22.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:44:22.615 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:44:22.985 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:44:22.985 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:44:23.015 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:44:23.056 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:44:23.126 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:44:23.126 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:44:23.168 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:44:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:44:23.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-21T23:44:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:44:23.341 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:44:24.440 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:46:24.497 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:46:24.538 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:46:25.615 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:46:25.638 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:46:25.638 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:46:25.638 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:46:25.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:46:26.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:46:26.033 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:46:26.388 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:46:26.389 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:46:26.417 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:46:26.465 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:46:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:46:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:46:26.585 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:46:26.620 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T23:46:26.622 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:46:26.624 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T23:46:26.663 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:46:27.736 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:48:27.790 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:48:27.830 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:48:28.922 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:48:28.939 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:48:28.939 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:48:28.940 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:48:28.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:48:29.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:48:29.189 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:48:29.522 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:48:29.522 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:48:29.555 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:48:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:48:29.674 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:48:29.675 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:48:29.715 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:48:29.742 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:48:29.752 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T23:48:29.778 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T23:48:29.780 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T23:48:29.875 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:48:30.923 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:50:30.977 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:50:31.015 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:50:32.918 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:50:32.936 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:50:32.936 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:50:32.937 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:50:32.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:50:33.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:50:33.208 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:50:33.515 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:50:33.516 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:50:33.549 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:50:33.587 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:50:33.659 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:50:33.659 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:50:33.699 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:50:33.729 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:50:33.737 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T23:50:33.765 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T23:50:33.765 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T23:50:33.792 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T23:50:33.832 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:50:34.914 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:52:34.969 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:52:35.011 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:52:36.159 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:52:36.177 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:52:36.178 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:52:36.178 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:52:36.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:52:36.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:52:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:52:36.761 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:52:36.761 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:52:36.792 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:52:36.832 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:52:36.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:52:36.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:52:36.946 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:52:36.974 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:52:36.986 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T23:52:36.986 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T23:52:36.986 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T23:52:36.987 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T23:52:37.025 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:52:38.171 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:54:38.233 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:54:38.272 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:54:39.345 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:54:39.362 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:54:39.363 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:54:39.363 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:54:39.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:54:39.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:54:39.622 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:54:39.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:54:39.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:54:39.983 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:54:40.023 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:54:40.093 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:54:40.093 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:54:40.132 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:54:40.163 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:54:40.168 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T23:54:40.168 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T23:54:40.168 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T23:54:40.169 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T23:54:40.169 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T23:54:40.169 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T23:54:40.170 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T23:54:40.226 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:54:41.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:56:41.328 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:56:41.367 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:56:42.436 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:56:42.455 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:56:42.455 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:56:42.455 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:56:42.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:56:42.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:56:42.722 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:56:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:56:43.047 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:56:43.078 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:56:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:56:43.187 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:56:43.187 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:56:43.221 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:56:43.257 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:56:43.264 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T23:56:43.265 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T23:56:43.265 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T23:56:43.265 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T23:56:43.265 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T23:56:43.265 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T23:56:43.267 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T23:56:43.305 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:56:44.319 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:58:44.377 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T23:58:44.417 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:58:45.476 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-21T23:58:45.493 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-21T23:58:45.493 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-21T23:58:45.493 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:58:45.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:58:45.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T23:58:45.752 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-21T23:58:46.074 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-21T23:58:46.075 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:58:46.106 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:58:46.146 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-21T23:58:46.217 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-21T23:58:46.217 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-21T23:58:46.260 INFO:teuthology.misc:Compressing logs... 2021-03-21T23:58:46.289 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-21T23:58:46.299 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-21T23:58:46.299 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-21T23:58:46.299 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-21T23:58:46.299 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-21T23:58:46.300 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-21T23:58:46.300 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-21T23:58:46.301 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-21T23:58:46.341 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-21T23:58:47.399 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:00:47.443 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:00:47.490 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:00:50.090 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:00:50.108 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:00:50.108 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:00:50.108 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:00:50.109 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:00:50.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:00:50.370 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:00:50.702 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:00:50.702 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:00:50.735 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:00:50.777 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:00:50.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:00:50.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:00:50.884 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:00:50.920 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:00:50.930 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T00:00:50.931 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T00:00:50.931 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T00:00:50.931 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T00:00:50.931 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T00:00:50.931 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T00:00:50.932 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:00:50.974 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:00:52.217 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:02:52.277 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:02:52.317 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:02:53.371 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:02:53.388 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:02:53.388 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:02:53.388 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:02:53.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:02:53.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:02:53.656 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:02:53.971 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:02:53.971 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:02:54.000 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:02:54.041 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:02:54.110 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:02:54.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:02:54.151 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:02:54.178 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:02:54.184 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T00:02:54.184 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T00:02:54.184 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T00:02:54.184 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T00:02:54.185 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T00:02:54.185 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T00:02:54.186 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:02:54.224 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:02:55.280 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:04:55.346 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:04:55.387 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:04:56.473 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:04:56.489 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:04:56.490 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:04:56.490 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:04:56.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:04:56.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:04:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:04:57.087 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:04:57.087 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:04:57.118 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:04:57.158 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:04:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:04:57.230 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:04:57.270 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:04:57.298 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:04:57.310 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T00:04:57.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T00:04:57.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T00:04:57.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T00:04:57.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T00:04:57.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T00:04:57.311 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:04:57.349 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:04:58.410 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:06:58.467 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:06:58.505 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:06:59.625 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:06:59.642 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:06:59.642 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:06:59.642 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:06:59.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:06:59.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:06:59.897 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:07:00.218 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:07:00.218 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:07:00.243 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:07:00.287 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:07:00.356 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:07:00.356 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:07:00.420 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:07:00.438 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T00:07:00.438 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T00:07:00.438 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T00:07:00.438 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T00:07:00.438 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T00:07:00.439 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T00:07:00.448 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:07:00.449 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:07:00.492 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:07:01.640 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:09:01.697 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:09:01.737 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:09:03.798 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:09:03.823 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:09:03.823 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:09:03.823 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:09:03.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:09:04.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:09:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:09:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:09:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:09:04.445 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:09:04.484 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:09:04.553 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:09:04.553 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:09:04.587 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:09:04.624 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:09:04.638 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T00:09:04.638 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T00:09:04.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T00:09:04.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T00:09:04.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T00:09:04.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T00:09:04.639 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:09:04.689 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:09:06.657 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:11:06.715 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:11:06.755 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:11:08.563 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:11:08.580 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:11:08.580 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:11:08.580 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:11:08.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:11:08.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:11:08.833 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:11:09.167 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:11:09.167 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:11:09.202 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:11:09.236 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:11:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:11:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:11:09.365 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:11:09.392 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:11:09.401 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T00:11:09.401 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T00:11:09.402 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T00:11:09.402 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T00:11:09.402 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T00:11:09.402 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dcc459d0-8a9e-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T00:11:09.404 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:11:09.445 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:11:10.457 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:13:10.511 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:13:10.550 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:13:11.649 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:13:11.667 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:13:11.667 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:13:11.667 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:13:11.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:13:11.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:13:11.914 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:13:14.950 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T00:13:14.988 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:13:16.027 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:15:16.089 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:15:16.129 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:15:17.250 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:15:17.274 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:15:17.274 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:15:17.274 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:15:17.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:15:17.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:15:17.551 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:15:20.578 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T00:15:20.619 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:15:21.660 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:17:21.716 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:17:21.788 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:17:23.548 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:17:23.564 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:17:23.564 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:17:23.564 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:17:23.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:17:23.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:17:23.817 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:17:26.626 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T00:17:26.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:17:27.657 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:19:27.712 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:19:27.753 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:19:28.862 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:19:28.879 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:19:28.879 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:19:28.880 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:19:28.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:19:29.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:19:29.129 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:19:29.325 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:19:29.358 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:19:29.358 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:19:29.401 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:19:29.401 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:19:29.403 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:19:29.432 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:19:29.502 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:19:29.502 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:19:29.551 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:19:29.579 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:19:29.610 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:19:29.646 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:19:29.647 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:19:29.672 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:19:29.673 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:19:29.711 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:19:30.777 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:21:30.835 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:21:30.898 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:21:32.057 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:21:32.074 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:21:32.074 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:21:32.074 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:21:32.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:21:32.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:21:32.346 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:21:32.531 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:21:32.563 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:21:32.563 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:21:32.600 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:21:32.600 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:21:32.601 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:21:32.634 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:21:32.703 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:21:32.703 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:21:32.752 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:21:32.779 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:21:32.800 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:21:32.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:21:32.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:21:32.874 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:21:32.874 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:21:32.914 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:21:33.985 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:23:34.052 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:23:34.121 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:23:35.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:23:35.210 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:23:35.210 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:23:35.210 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:23:35.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:23:35.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:23:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:23:35.669 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:23:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:23:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:23:35.735 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:23:35.735 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:23:35.736 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:23:35.769 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:23:35.801 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:23:35.802 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:23:35.893 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:23:36.920 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:25:36.978 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:25:37.020 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:25:38.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:25:38.091 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:25:38.091 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:25:38.092 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:25:38.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:25:38.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:25:38.348 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:25:38.544 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:25:38.574 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:25:38.574 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:25:38.611 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:25:38.611 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:25:38.612 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:25:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:25:38.678 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:25:38.680 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:25:38.723 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:25:39.844 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:27:39.901 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:27:39.947 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:27:41.044 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:27:41.061 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:27:41.061 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:27:41.061 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:27:41.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:27:41.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:27:41.330 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:27:41.543 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:27:41.543 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:27:41.582 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:27:41.615 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:27:41.647 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:27:41.648 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:27:41.686 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:27:42.826 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:29:42.885 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:29:42.928 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:29:44.030 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:29:44.048 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:29:44.048 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:29:44.048 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:29:44.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:29:44.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:29:44.304 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:29:44.536 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:29:44.536 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:29:44.577 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:29:44.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:29:44.643 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:29:44.645 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:29:44.690 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:29:45.805 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:31:45.860 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:31:45.899 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:31:46.990 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:31:47.008 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:31:47.008 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:31:47.009 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:31:47.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:31:47.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:31:47.266 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:31:47.504 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:31:47.504 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:31:47.545 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:31:47.574 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:31:47.611 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:31:47.612 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:31:47.651 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:31:48.720 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:33:48.775 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:33:48.818 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:33:49.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:33:49.963 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:33:49.963 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:33:49.964 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:33:49.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:33:50.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:33:50.236 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:33:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:33:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:33:50.487 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:33:50.518 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:33:50.554 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:33:50.555 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:33:50.600 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:33:51.658 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:35:51.716 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:35:51.762 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:35:52.888 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:35:52.906 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:35:52.906 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:35:52.906 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:35:52.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:35:53.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:35:53.161 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:36:27.715 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T00:36:27.754 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:36:28.806 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:38:28.860 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:38:28.898 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:38:29.970 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:38:29.987 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:38:29.987 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:38:29.987 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:38:29.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:38:30.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:38:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:38:30.311 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T00:38:31.415 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T00:38:32.476 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T00:38:33.573 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T00:38:34.669 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T00:38:35.767 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T00:38:36.867 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T00:38:37.965 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T00:38:39.068 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T00:38:40.125 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T00:38:40.126 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T00:38:40.163 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:38:41.196 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:40:41.253 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:40:41.294 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:40:42.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:40:42.380 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:40:42.380 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:40:42.380 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:40:42.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:40:42.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:40:42.651 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:41:15.221 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:41:15.255 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:41:15.255 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:41:15.303 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:41:15.304 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:41:15.305 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:41:15.330 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:41:15.334 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:41:15.404 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:41:15.404 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:41:15.428 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:41:15.428 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:41:15.429 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:41:15.473 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:41:15.492 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:41:15.543 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:41:15.543 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:41:15.569 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:41:15.569 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:41:15.612 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:41:16.619 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:43:16.677 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:43:16.716 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:43:17.820 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:43:17.837 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:43:17.837 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:43:17.837 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:43:17.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:43:18.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:43:18.092 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:43:18.290 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:43:18.321 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:43:18.321 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:43:18.359 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:43:18.359 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:43:18.360 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:43:18.390 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:43:18.425 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:43:18.460 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:43:18.460 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:43:18.492 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:43:18.492 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:43:18.493 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:43:18.530 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:43:18.557 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:43:18.598 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:43:18.598 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:43:18.623 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:43:18.623 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:43:18.666 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:43:19.679 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:45:19.736 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:45:19.777 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:45:20.840 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:45:20.858 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:45:20.858 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:45:20.858 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:45:20.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:45:21.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:45:21.112 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:45:21.304 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:45:21.335 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:45:21.335 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:45:21.371 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:45:21.372 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:45:21.374 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:45:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:45:21.439 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:45:21.478 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:45:21.478 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:45:21.508 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:45:21.508 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:45:21.509 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:45:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:45:21.573 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:45:21.619 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:45:21.619 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:45:21.643 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:45:21.643 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:45:21.681 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:45:22.746 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:47:22.815 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:47:22.858 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:47:23.897 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:47:23.914 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:47:23.914 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:47:23.915 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:47:23.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:47:24.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:47:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:47:24.363 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:47:24.393 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:47:24.393 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:47:24.429 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:47:24.429 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:47:24.431 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:47:24.463 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:47:24.532 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T00:47:24.532 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:47:24.583 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:47:24.609 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:47:24.629 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T00:47:24.677 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:47:24.677 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:47:24.701 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T00:47:24.702 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:47:24.741 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:47:25.782 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:49:25.838 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:49:25.875 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:49:26.885 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:49:26.902 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:49:26.902 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:49:26.903 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:49:26.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:49:27.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:49:27.151 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:49:27.344 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:49:27.374 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:49:27.374 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:49:27.411 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:49:27.411 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:49:27.412 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:49:27.444 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:49:27.477 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:49:27.479 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:49:27.516 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:49:28.525 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:51:28.592 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:51:28.631 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:51:29.711 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:51:29.728 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:51:29.729 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:51:29.729 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:51:29.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:51:29.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:51:29.982 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:51:30.188 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:51:30.218 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:51:30.218 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:51:30.254 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:51:30.255 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:51:30.256 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:51:30.289 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:51:30.322 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:51:30.323 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:51:30.361 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:51:31.412 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:53:31.469 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:53:31.510 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:53:33.517 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:53:33.536 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:53:33.536 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:53:33.536 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:53:33.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:53:33.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:53:33.806 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:53:34.001 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:53:34.031 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:53:34.031 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:53:34.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:53:34.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:53:34.070 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:53:34.100 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:53:34.137 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:53:34.138 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:53:34.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:53:35.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:55:35.581 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:55:35.625 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:55:37.395 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:55:37.413 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:55:37.413 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:55:37.413 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:55:37.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:55:37.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:55:37.665 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:55:37.882 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:55:37.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:55:37.922 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:55:37.954 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:55:37.988 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:55:37.989 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:55:38.029 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:55:39.132 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:57:39.186 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:57:39.233 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:57:40.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:57:40.485 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:57:40.485 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:57:40.486 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:57:40.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:57:40.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:57:40.740 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:57:40.945 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:57:40.945 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:57:40.984 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:57:41.015 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:57:41.051 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:57:41.052 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:57:41.090 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:57:42.163 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:59:42.225 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T00:59:42.335 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:59:44.276 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T00:59:44.293 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T00:59:44.293 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T00:59:44.293 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:59:44.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:59:44.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T00:59:44.569 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T00:59:44.746 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T00:59:44.746 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T00:59:44.784 INFO:teuthology.misc:Compressing logs... 2021-03-22T00:59:44.814 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T00:59:44.848 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T00:59:44.850 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T00:59:44.891 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T00:59:45.941 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:01:45.995 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:01:46.055 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:01:47.901 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:01:47.927 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:01:47.927 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:01:47.928 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:01:47.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:01:48.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:01:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:01:48.383 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:01:48.383 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:01:48.421 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:01:48.455 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:01:48.488 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T01:01:48.489 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T01:01:48.529 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:01:49.625 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:03:49.684 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:03:49.723 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:03:50.764 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:03:50.781 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:03:50.781 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:03:50.781 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:03:50.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:03:51.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:03:51.037 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:03:51.245 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:03:51.245 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:03:51.288 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:03:51.316 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:03:51.354 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T01:03:51.357 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T01:03:51.394 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:03:52.451 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:05:52.508 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:05:52.556 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:05:53.636 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:05:53.653 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:05:53.653 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:05:53.654 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:05:53.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:05:53.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:05:53.925 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:05:56.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T01:05:56.973 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:05:58.109 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:07:58.166 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:07:58.204 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:08:00.175 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:08:00.193 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:08:00.193 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:08:00.193 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:08:00.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:08:00.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:08:00.445 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:08:00.474 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:08:01.572 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:08:02.669 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:08:03.767 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:08:04.865 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:08:05.966 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:08:07.067 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:08:08.125 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:08:09.188 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:08:10.318 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:08:10.318 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T01:08:10.358 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:08:11.484 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:10:11.545 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:10:11.592 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:10:13.427 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:10:13.443 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:10:13.444 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:10:13.444 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:10:13.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:10:13.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:10:13.768 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:10:14.579 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:10:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:10:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:10:14.683 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:10:14.683 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:10:14.685 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:10:14.717 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:10:14.720 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:10:14.803 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:10:14.803 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:10:14.832 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:10:14.832 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:10:14.833 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:10:14.876 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:10:14.902 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:10:14.949 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:10:14.949 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:10:14.979 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:10:14.979 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:10:15.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:10:16.126 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:12:16.184 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:12:16.223 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:12:18.205 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:12:18.222 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:12:18.222 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:12:18.222 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:12:18.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:12:18.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:12:18.497 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:12:18.679 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:12:18.709 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:12:18.709 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:12:18.753 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:12:18.754 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:12:18.755 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:12:18.783 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:12:18.819 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:12:18.853 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:12:18.853 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:12:18.884 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:12:18.884 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:12:18.885 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:12:18.922 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:12:18.949 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:12:18.990 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:12:18.990 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:12:19.014 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:12:19.015 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:12:19.064 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:12:20.135 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:14:20.202 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:14:20.245 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:14:22.037 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:14:22.057 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:14:22.057 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:14:22.057 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:14:22.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:14:22.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:14:22.304 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:14:22.504 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:14:22.535 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:14:22.535 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:14:22.573 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:14:22.573 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:14:22.574 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:14:22.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:14:22.639 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:14:22.683 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:14:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:14:22.709 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:14:22.709 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:14:22.710 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:14:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:14:22.776 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:14:22.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:14:22.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:14:22.847 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:14:22.847 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:14:22.886 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:14:23.905 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:16:23.959 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:16:23.999 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:16:26.139 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:16:26.165 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:16:26.165 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:16:26.166 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:16:26.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:16:26.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:16:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:16:26.616 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:16:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:16:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:16:26.681 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:16:26.682 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:16:26.682 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:16:26.720 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:16:26.793 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:16:26.793 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:16:26.835 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:16:26.863 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:16:26.891 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:16:26.932 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:16:26.932 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:16:26.957 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:16:26.958 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:16:27.000 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:16:28.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:18:28.795 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:18:28.839 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:18:31.015 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:18:31.036 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:18:31.036 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:18:31.036 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:18:31.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:18:31.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:18:31.321 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:18:31.499 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:18:31.527 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:18:31.527 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:18:31.564 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:18:31.565 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:18:31.566 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:18:31.595 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:18:31.630 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T01:18:31.632 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T01:18:31.674 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:18:33.014 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:20:33.074 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:20:33.131 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:20:35.153 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:20:35.173 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:20:35.173 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:20:35.173 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:20:35.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:20:35.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:20:35.452 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:20:35.653 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:20:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:20:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:20:35.729 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:20:35.730 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:20:35.731 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:20:35.759 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:20:35.800 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T01:20:35.801 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T01:20:35.864 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:20:37.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:22:37.877 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:22:37.916 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:22:39.683 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:22:39.700 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:22:39.700 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:22:39.700 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:22:39.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:22:39.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:22:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:22:40.188 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:22:40.188 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:22:40.216 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:22:40.257 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:22:40.281 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T01:22:40.282 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T01:22:40.323 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:22:41.385 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:24:41.443 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:24:41.482 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:24:42.560 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:24:42.578 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:24:42.578 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:24:42.579 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:24:42.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:24:42.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:24:42.845 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:24:43.040 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:24:43.040 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:24:43.079 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:24:43.109 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:24:43.145 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T01:24:43.147 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T01:24:43.208 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:24:44.307 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:26:44.360 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:26:44.409 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:26:45.504 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:26:45.524 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:26:45.525 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:26:45.525 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:26:45.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:26:45.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:26:45.775 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:26:45.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:26:45.980 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:26:46.020 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:26:46.050 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:26:46.086 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T01:26:46.088 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T01:26:46.126 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:26:47.188 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:28:47.253 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:28:47.294 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:28:48.387 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:28:48.405 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:28:48.405 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:28:48.406 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:28:48.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:28:48.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:28:48.657 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:28:48.873 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:28:48.873 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:28:48.914 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:28:48.944 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:28:48.982 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T01:28:48.983 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T01:28:49.026 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:28:50.101 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:30:50.158 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:30:50.197 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:30:51.266 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:30:51.283 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:30:51.283 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:30:51.283 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:30:51.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:30:51.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:30:51.555 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:30:51.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:30:51.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:30:51.774 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:30:51.814 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:30:51.840 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T01:30:51.842 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T01:30:51.882 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:30:52.931 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:32:52.986 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:32:53.026 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:32:54.082 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:32:54.107 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:32:54.107 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:32:54.107 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:32:54.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:32:54.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:32:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:33:12.771 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T01:33:12.873 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:33:13.991 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:35:14.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:35:14.084 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:35:15.322 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:35:15.340 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:35:15.340 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:35:15.340 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:35:15.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:35:15.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:35:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:35:15.765 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:35:16.832 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:35:17.891 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:35:18.994 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:35:20.094 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:35:21.211 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:35:22.312 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:35:23.409 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:35:24.506 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:35:25.603 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:35:25.603 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T01:35:25.643 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:35:26.659 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:37:26.716 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:37:26.755 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:37:28.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:37:28.705 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:37:28.705 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:37:28.705 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:37:28.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:37:28.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:37:28.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T01:37:28.968 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:37:29.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:39:30.050 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:39:30.092 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:39:32.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:39:32.257 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:39:32.257 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:39:32.258 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:39:32.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:39:32.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:39:32.512 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:39:32.668 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T01:39:32.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:39:32.711 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:39:32.737 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:39:32.737 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:39:32.739 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:39:32.771 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T01:39:32.781 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:39:32.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:39:32.850 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:39:32.867 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:39:32.867 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:39:32.870 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:39:32.918 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:39:32.925 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T01:39:32.986 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:39:32.987 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:39:33.004 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:39:33.004 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:39:33.046 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:39:34.924 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:41:34.979 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:41:35.018 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:41:37.098 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:41:37.115 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:41:37.116 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:41:37.116 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:41:37.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:41:37.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:41:37.374 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:41:37.506 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T01:41:37.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:41:37.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:41:37.552 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:41:37.553 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:41:37.555 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:41:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:41:37.675 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:41:37.676 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:41:37.708 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:41:37.735 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T01:41:37.743 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:41:37.811 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:41:37.811 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:41:37.825 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:41:37.826 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:41:37.881 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:41:39.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:43:39.289 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:43:39.331 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:43:41.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:43:41.210 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:43:41.211 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:43:41.211 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:43:41.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:43:41.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:43:41.482 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:43:41.629 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T01:43:41.666 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:43:41.666 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:43:41.683 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:43:41.683 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:43:41.686 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:43:41.736 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:43:41.807 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:43:41.807 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:43:41.848 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:43:41.871 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T01:43:41.880 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:43:41.948 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:43:41.949 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:43:41.971 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:43:41.972 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:43:42.014 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:43:43.092 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:45:43.152 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:45:43.193 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:45:44.276 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:45:44.293 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:45:44.293 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:45:44.293 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:45:44.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:45:44.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:45:44.545 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:45:44.734 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:45:44.735 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:45:44.756 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:45:44.803 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:45:44.872 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:45:44.872 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:45:44.911 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:45:44.952 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:45:45.021 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:45:45.021 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:45:45.083 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:45:46.131 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:47:46.195 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:47:46.234 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:47:47.389 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:47:47.406 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:47:47.406 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:47:47.406 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:47:47.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:47:47.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:47:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:47:47.855 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:47:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:47:47.883 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:47:47.927 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:47:47.997 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:47:47.997 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:47:48.033 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:47:48.065 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:47:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:47:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:47:48.198 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:47:49.372 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:49:49.435 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:49:49.475 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:49:51.311 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:49:51.330 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:49:51.330 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:49:51.330 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:49:51.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:49:51.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:49:51.605 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:49:51.784 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:49:51.784 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:49:51.804 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:49:51.853 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:49:51.922 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:49:51.922 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:49:51.958 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:49:51.992 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:49:52.040 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping 2021-03-22T01:49:52.044 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.b.log: file size changed while zipping 2021-03-22T01:49:52.100 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping 2021-03-22T01:49:52.192 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2021-03-22T01:49:52.260 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2021-03-22T01:49:52.388 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping 2021-03-22T01:49:52.443 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2021-03-22T01:49:52.476 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:49:52.476 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:49:52.818 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:49:53.912 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:51:53.968 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:51:54.009 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:51:55.059 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:51:55.076 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:51:55.076 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:51:55.077 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:51:55.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:51:55.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:51:55.328 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:52:13.766 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T01:52:13.808 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:52:14.828 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:54:14.893 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:54:14.929 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:54:15.925 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:54:15.941 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:54:15.942 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:54:15.942 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:54:15.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:54:16.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:54:16.198 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:54:16.226 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:54:17.323 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:54:18.419 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:54:19.515 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:54:20.616 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:54:21.674 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:54:22.772 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:54:23.870 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:54:24.968 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:54:26.065 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T01:54:26.065 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T01:54:26.102 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:54:27.107 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:56:27.167 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:56:27.204 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:56:28.296 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:56:28.332 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:56:28.332 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:56:28.332 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:56:28.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:56:28.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:56:28.624 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:56:29.290 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:56:29.331 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:56:29.331 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:56:29.374 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:56:29.374 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:56:29.375 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:56:29.404 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:56:29.438 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:56:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:56:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:56:29.502 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:56:29.502 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:56:29.503 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:56:29.543 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:56:29.566 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:56:29.611 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:56:29.612 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:56:29.636 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:56:29.636 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:56:29.675 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:56:30.713 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:58:30.771 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T01:58:30.814 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:58:32.676 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T01:58:32.694 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T01:58:32.694 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T01:58:32.694 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:58:32.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:58:32.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T01:58:32.955 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:58:33.190 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T01:58:33.222 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:58:33.222 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:58:33.254 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T01:58:33.255 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:58:33.257 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:58:33.293 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:58:33.320 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T01:58:33.364 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:58:33.364 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:58:33.390 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T01:58:33.390 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:58:33.392 INFO:teuthology.misc:Compressing logs... 2021-03-22T01:58:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:58:33.456 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T01:58:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:58:33.507 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:58:33.531 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T01:58:33.532 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T01:58:33.571 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T01:58:34.676 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:00:34.753 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:00:34.804 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:00:36.006 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:00:36.025 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:00:36.025 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:00:36.025 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:00:36.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:00:36.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:00:36.285 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:00:37.045 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:00:37.080 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:00:37.080 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:00:37.117 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:00:37.117 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:00:37.118 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:00:37.150 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:00:37.187 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:00:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:00:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:00:37.256 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:00:37.256 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:00:37.257 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:00:37.295 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:00:37.324 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:00:37.365 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:00:37.366 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:00:37.393 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:00:37.394 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:00:37.435 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:00:38.670 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:02:38.732 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:02:38.773 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:02:40.663 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:02:40.699 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:02:40.700 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:02:40.700 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:02:40.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:02:40.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:02:40.987 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:02:41.173 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:02:41.204 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:02:41.204 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:02:41.240 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:02:41.241 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:02:41.242 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:02:41.274 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:02:41.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:02:41.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:02:41.395 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:02:41.423 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:02:41.452 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:02:41.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:02:41.492 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:02:41.520 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:02:41.520 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:02:41.561 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:02:43.165 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:04:43.225 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:04:43.267 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:04:45.138 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:04:45.160 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:04:45.160 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:04:45.160 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:04:45.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:04:45.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:04:45.417 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:04:45.630 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:04:45.660 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:04:45.660 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:04:45.698 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:04:45.698 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:04:45.699 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:04:45.731 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:04:45.765 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T02:04:45.767 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T02:04:45.807 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:04:47.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:06:47.529 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:06:47.569 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:06:49.458 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:06:49.475 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:06:49.475 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:06:49.475 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:06:49.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:06:49.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:06:49.730 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:06:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:06:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:06:49.990 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:06:50.030 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:06:50.057 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T02:06:50.058 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T02:06:50.154 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:06:51.774 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:08:51.832 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:08:51.872 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:08:53.673 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:08:53.690 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:08:53.690 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:08:53.690 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:08:53.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:08:53.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:08:53.965 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:08:54.164 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:08:54.164 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:08:54.210 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:08:54.242 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:08:54.279 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T02:08:54.280 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T02:08:54.322 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:08:55.379 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:10:55.437 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:10:55.480 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:10:56.602 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:10:56.628 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:10:56.629 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:10:56.629 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:10:56.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:10:56.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:10:56.903 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:10:57.104 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:10:57.104 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:10:57.146 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:10:57.173 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:10:57.211 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T02:10:57.213 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T02:10:57.253 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:10:58.417 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:12:58.487 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:12:58.527 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:12:59.615 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:12:59.633 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:12:59.634 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:12:59.634 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:12:59.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:12:59.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:12:59.895 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:13:00.098 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:13:00.099 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:13:00.136 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:13:00.170 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:13:00.200 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T02:13:00.202 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T02:13:00.242 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:13:01.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:15:01.464 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:15:01.503 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:15:02.619 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:15:02.638 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:15:02.639 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:15:02.639 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:15:02.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:15:02.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:15:02.909 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:15:21.286 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T02:15:21.325 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:15:22.451 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:17:22.516 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:17:22.555 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:17:23.620 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:17:23.637 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:17:23.638 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:17:23.638 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:17:23.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:17:23.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:17:23.898 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:17:26.946 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T02:17:26.985 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:17:27.998 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:19:28.056 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:19:28.095 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:19:29.157 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:19:29.174 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:19:29.174 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:19:29.174 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:19:29.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:19:29.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:19:29.428 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:19:32.450 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T02:19:32.487 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:19:33.489 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:21:33.545 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:21:33.586 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:21:34.630 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:21:34.648 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:21:34.648 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:21:34.648 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:21:34.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:21:34.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:21:34.920 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:21:35.380 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T02:21:35.413 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:21:35.413 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:21:35.438 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:21:35.439 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:21:35.440 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:21:35.483 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:21:35.497 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T02:21:35.552 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:21:35.552 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:21:35.572 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:21:35.572 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:21:35.573 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:21:35.620 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:21:35.629 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T02:21:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:21:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:21:35.705 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:21:35.705 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:21:35.742 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:21:36.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:23:36.814 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:23:36.854 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:23:37.927 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:23:37.944 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:23:37.944 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:23:37.944 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:23:37.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:23:38.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:23:38.202 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:23:38.498 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T02:23:38.531 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:23:38.531 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:23:38.557 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:23:38.557 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:23:38.558 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:23:38.602 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:23:38.613 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T02:23:38.673 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:23:38.673 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:23:38.688 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:23:38.688 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:23:38.690 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:23:38.743 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:23:38.746 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T02:23:38.813 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:23:38.814 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:23:38.829 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:23:38.829 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:23:38.871 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:23:39.952 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:25:40.011 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:25:40.049 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:25:41.094 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:25:41.112 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:25:41.112 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:25:41.112 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:25:41.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:25:41.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:25:41.371 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:26:12.803 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T02:26:12.842 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:26:13.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:28:13.907 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:28:13.948 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:28:16.181 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:28:16.211 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:28:16.212 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:28:16.212 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:28:16.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:28:16.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:28:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:28:16.760 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T02:28:16.791 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:28:16.791 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:28:16.810 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:28:16.811 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:28:16.812 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:28:16.863 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:28:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:28:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:28:16.967 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:28:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:28:17.006 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T02:28:17.073 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:28:17.073 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:28:17.082 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:28:17.082 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:28:17.122 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:28:18.920 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:30:18.980 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:30:19.020 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:30:20.873 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:30:20.890 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:30:20.890 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:30:20.890 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:30:20.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:30:21.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:30:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:30:21.454 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:30:21.454 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:30:21.481 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:30:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:30:21.592 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:30:21.593 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:30:21.634 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:30:21.668 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:30:21.742 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:30:21.742 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:30:21.801 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:30:22.844 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:32:22.900 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:32:22.969 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:32:25.183 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:32:25.200 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:32:25.200 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:32:25.200 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:32:25.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:32:25.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:32:25.469 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:32:25.791 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:32:25.791 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:32:25.824 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:32:25.862 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:32:25.934 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:32:25.934 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:32:25.975 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:32:26.006 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:32:26.078 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:32:26.079 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:32:26.138 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:32:28.215 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:34:28.272 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:34:28.314 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:34:30.064 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:34:30.082 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:34:30.082 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:34:30.083 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:34:30.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:34:30.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:34:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:34:30.672 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:34:30.673 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:34:30.705 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:34:30.742 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:34:30.812 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:34:30.812 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:34:30.857 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:34:30.884 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:34:30.953 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:34:30.953 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:34:31.035 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:34:32.073 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:36:32.130 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:38:27.215 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:38:29.061 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:38:29.079 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:38:29.079 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:38:29.079 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:38:29.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:38:29.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:38:29.431 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:38:29.829 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:38:29.829 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:38:29.863 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:38:29.923 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:38:29.996 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:38:29.996 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:38:30.063 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:38:30.096 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:38:30.174 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:38:30.174 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:38:30.307 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:38:31.448 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:40:31.511 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:40:31.550 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:40:33.533 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:40:33.550 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:40:33.550 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:40:33.550 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:40:33.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:40:33.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:40:33.822 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:40:34.135 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:40:34.135 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:40:34.166 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:40:34.210 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:40:34.285 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:40:34.286 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:40:34.325 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:40:34.356 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:40:34.426 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:40:34.426 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:40:34.498 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:40:35.790 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:42:35.846 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:42:35.889 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:42:37.782 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:42:37.799 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:42:37.799 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:42:37.799 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:42:37.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:42:38.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:42:38.052 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:42:38.383 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:42:38.384 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:42:38.418 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:42:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:42:38.539 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:42:38.540 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:42:38.579 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:42:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:42:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:42:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:42:38.754 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:42:40.286 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:44:40.365 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:44:40.403 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:44:42.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:44:42.211 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:44:42.212 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:44:42.212 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:44:42.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:44:42.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:44:42.470 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:44:42.807 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:44:42.807 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:44:42.836 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:44:42.877 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:44:42.948 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:44:42.948 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:44:42.987 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:44:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:44:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:44:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:44:43.222 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:44:44.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:46:44.399 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:46:44.438 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:46:46.378 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:46:46.395 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:46:46.395 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:46:46.396 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:46:46.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:46:46.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:46:46.701 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:46:47.049 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:46:47.049 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:46:47.080 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:46:47.123 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:46:47.200 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:46:47.200 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:46:47.292 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:46:47.340 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:46:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:46:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:46:47.533 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:46:48.760 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:48:48.823 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:48:48.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:48:50.707 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:48:50.725 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:48:50.726 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:48:50.726 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:48:50.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:48:50.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:48:50.986 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:48:51.322 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:48:51.322 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:48:51.354 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:48:51.392 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:48:51.462 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:48:51.462 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:48:51.501 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:48:51.531 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:48:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:48:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:48:51.673 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:48:52.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:50:52.874 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:50:52.912 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:50:53.979 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:50:53.999 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:50:53.999 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:50:54.000 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:50:54.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:50:54.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:50:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:50:54.598 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:50:54.598 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:50:54.630 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:50:54.669 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:50:54.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:50:54.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:50:54.774 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:50:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:50:54.877 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:50:54.878 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:50:54.962 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:50:56.039 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:52:56.100 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:52:56.192 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:52:58.334 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:52:58.359 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:52:58.359 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:52:58.359 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:52:58.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:52:58.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:52:58.641 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:52:58.954 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T02:52:58.954 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:52:58.986 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:52:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T02:52:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T02:52:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:52:59.146 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:52:59.176 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T02:52:59.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T02:52:59.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T02:52:59.312 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:53:00.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:55:01.020 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:55:01.062 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:55:03.184 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:55:03.203 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:55:03.203 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:55:03.203 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:55:03.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:55:03.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:55:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:55:06.498 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T02:55:06.541 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:55:08.381 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:57:08.442 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T02:57:08.481 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:57:10.571 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T02:57:10.589 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T02:57:10.589 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T02:57:10.589 INFO:teuthology.misc:Compressing logs... 2021-03-22T02:57:10.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:57:10.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T02:57:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T02:57:10.918 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T02:57:11.977 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T02:57:13.076 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T02:57:14.173 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T02:57:15.271 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T02:57:16.374 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T02:57:17.475 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T02:57:18.574 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T02:57:19.674 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T02:58:20.678 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T02:58:20.721 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T02:58:21.792 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:00:21.858 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:00:21.899 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:00:23.002 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:00:23.020 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:00:23.020 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:00:23.020 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:00:23.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:00:23.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:00:23.291 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:00:23.463 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T03:00:23.493 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:00:23.493 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:00:23.509 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:00:23.510 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:00:23.511 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:00:23.559 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T03:00:23.562 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:00:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:00:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:00:23.650 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:00:23.651 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:00:23.652 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:00:23.689 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T03:00:23.699 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:00:23.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T03:00:23.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:00:23.784 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T03:00:23.785 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:00:23.825 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:00:24.959 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:02:25.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:02:25.065 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:02:26.179 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:02:26.196 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:02:26.197 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:02:26.197 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:02:26.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:02:26.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:02:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:02:26.678 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T03:02:26.713 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:02:26.713 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:02:26.741 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:02:26.742 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:02:26.744 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:02:26.782 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:02:26.857 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:02:26.857 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:02:26.900 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:02:26.920 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T03:02:26.928 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:02:26.999 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T03:02:26.999 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:02:27.017 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T03:02:27.017 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:02:27.059 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:02:28.138 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:04:28.197 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:04:28.243 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:04:29.420 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:04:29.438 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:04:29.438 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:04:29.438 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:04:29.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:04:29.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:04:29.708 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:04:29.857 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T03:04:29.893 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:04:29.894 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:04:29.911 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:04:29.911 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:04:29.914 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:04:29.974 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:04:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:04:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:04:30.081 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:04:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:04:30.118 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T03:04:30.180 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T03:04:30.180 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:04:30.199 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T03:04:30.199 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:04:30.249 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:04:31.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:06:31.397 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:06:31.436 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:06:32.502 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:06:32.518 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:06:32.519 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:06:32.519 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:06:32.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:06:32.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:06:32.790 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:06:32.933 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T03:06:32.969 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:06:32.969 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:06:32.999 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:06:32.999 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:06:33.000 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:06:33.040 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:06:33.110 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:06:33.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:06:33.156 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:06:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:06:33.192 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T03:06:33.262 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T03:06:33.263 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:06:33.283 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T03:06:33.283 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:06:33.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:06:34.361 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:08:34.418 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:08:34.463 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:08:36.617 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:08:36.635 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:08:36.635 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:08:36.636 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:08:36.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:08:36.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:08:36.914 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:08:37.094 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:08:37.094 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:08:37.113 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:08:37.169 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:08:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:08:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:08:37.274 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:08:37.322 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:08:37.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T03:08:37.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:08:37.457 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:08:38.834 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:10:38.895 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:10:38.935 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:10:40.034 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:10:40.051 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:10:40.051 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:10:40.052 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:10:40.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:10:40.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:10:40.305 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:10:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:10:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:10:40.515 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:10:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:10:40.640 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:10:40.640 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:10:40.675 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:10:40.704 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T03:10:40.716 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:10:40.793 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:10:40.834 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:10:41.913 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:12:41.976 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:12:42.015 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:12:43.051 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:12:43.068 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:12:43.068 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:12:43.068 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:12:43.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:12:43.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:12:43.341 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:12:43.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:12:43.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:12:43.508 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:12:43.559 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:12:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:12:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:12:43.666 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:12:43.687 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T03:12:43.687 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T03:12:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:12:43.797 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-03-22T03:12:43.797 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-mgr.a.log.gz already exists; not overwritten 2021-03-22T03:12:43.797 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-osd.1.log.gz already exists; not overwritten 2021-03-22T03:12:43.799 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:12:43.837 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:12:44.853 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:14:44.909 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:14:44.950 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:14:46.002 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:14:46.019 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:14:46.019 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:14:46.019 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:14:46.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:14:46.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:14:46.276 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:14:46.460 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:14:46.461 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:14:46.480 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:14:46.532 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:14:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:14:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:14:46.646 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:14:46.687 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:14:46.688 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T03:14:46.688 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T03:14:46.688 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-mds.a.smithi205.pwovqt.log.gz already exists; not overwritten 2021-03-22T03:14:46.689 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-mgr.a.log.gz already exists; not overwritten 2021-03-22T03:14:46.689 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-osd.1.log.gz already exists; not overwritten 2021-03-22T03:14:46.695 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:14:46.734 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:14:47.802 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:16:47.863 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:16:47.902 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:16:48.944 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:16:48.961 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:16:48.962 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:16:48.962 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:16:48.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:16:49.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:16:49.214 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:16:49.390 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:16:49.391 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:16:49.404 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:16:49.460 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:16:49.529 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:16:49.529 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:16:49.567 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:16:49.587 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T03:16:49.587 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T03:16:49.588 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-mds.a.smithi205.pwovqt.log.gz already exists; not overwritten 2021-03-22T03:16:49.588 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-mgr.a.log.gz already exists; not overwritten 2021-03-22T03:16:49.588 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-osd.1.log.gz already exists; not overwritten 2021-03-22T03:16:49.597 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:16:49.600 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:16:49.639 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:16:50.663 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:18:50.727 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:18:50.764 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:18:51.771 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:18:51.789 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:18:51.789 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:18:51.789 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:18:51.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:18:52.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:18:52.060 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:18:52.279 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:18:52.279 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:18:52.294 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:18:52.350 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:18:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:18:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:18:52.460 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:18:52.482 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T03:18:52.482 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T03:18:52.482 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-mds.a.smithi205.pwovqt.log.gz already exists; not overwritten 2021-03-22T03:18:52.483 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-mgr.a.log.gz already exists; not overwritten 2021-03-22T03:18:52.483 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-osd.1.log.gz already exists; not overwritten 2021-03-22T03:18:52.483 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-mds.nfs-cephfs.smithi205.dujlnw.log.gz already exists; not overwritten 2021-03-22T03:18:52.489 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:18:52.492 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:18:52.527 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:18:53.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:20:53.558 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:20:53.678 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:20:54.765 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:20:54.781 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:20:54.781 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:20:54.781 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:20:54.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:20:55.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:20:55.032 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:20:55.189 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:20:55.189 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:20:55.213 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:20:55.258 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:20:55.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:20:55.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:20:55.359 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:20:55.385 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T03:20:55.385 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T03:20:55.385 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-mds.a.smithi205.pwovqt.log.gz already exists; not overwritten 2021-03-22T03:20:55.385 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-mgr.a.log.gz already exists; not overwritten 2021-03-22T03:20:55.385 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-osd.1.log.gz already exists; not overwritten 2021-03-22T03:20:55.385 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/d2e1deac-8abb-11eb-90a1-001a4aab830c/ceph-mds.nfs-cephfs.smithi205.dujlnw.log.gz already exists; not overwritten 2021-03-22T03:20:55.395 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:20:55.399 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:20:55.435 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:20:56.484 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:22:56.539 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:22:56.576 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:22:57.632 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:22:57.648 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:22:57.648 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:22:57.648 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:22:57.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:22:57.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:22:57.901 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:23:16.291 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T03:23:16.331 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:23:17.386 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:25:17.445 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:25:17.483 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:25:18.508 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:25:18.525 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:25:18.526 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:25:18.526 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:25:18.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:25:18.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:25:18.796 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:25:21.830 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T03:25:21.870 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:25:22.868 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:27:22.924 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:27:22.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:27:24.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:27:24.720 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:27:24.721 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:27:24.721 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:27:24.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:27:24.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:27:24.974 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:27:28.006 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T03:27:28.047 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:27:29.165 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:29:29.220 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:29:29.259 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:29:31.092 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:29:31.109 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:29:31.109 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:29:31.109 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:29:31.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:29:31.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:29:31.361 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:29:34.402 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T03:29:34.441 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:29:36.209 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:31:36.266 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:40:05.489 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:40:07.478 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:40:07.960 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:40:07.960 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:40:07.960 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:40:07.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:40:08.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:40:08.234 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:40:08.416 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:40:08.446 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:40:08.446 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:40:08.491 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:40:08.491 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:40:08.493 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:40:08.518 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:40:08.589 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T03:40:08.589 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:40:08.646 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:40:08.672 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:40:08.694 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T03:40:08.740 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T03:40:08.742 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:40:08.766 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T03:40:08.767 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:40:08.806 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:40:10.478 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:42:10.535 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:42:10.574 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:42:12.402 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:42:12.419 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:42:12.419 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:42:12.419 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:42:12.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:42:12.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:42:12.676 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:42:12.868 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:42:12.898 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:42:12.898 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:42:12.934 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:42:12.934 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:42:12.936 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:42:12.968 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:42:13.001 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T03:42:13.003 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:42:13.044 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:42:14.172 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:44:14.238 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:44:14.292 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:44:16.220 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:44:16.255 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:44:16.255 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:44:16.255 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:44:16.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:44:16.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:44:16.508 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:44:16.713 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:44:16.749 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:44:16.750 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:44:16.785 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:44:16.785 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:44:16.787 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:44:16.823 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:44:16.857 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T03:44:16.861 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:44:16.902 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:44:18.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:46:18.418 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:46:18.463 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:46:20.298 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:46:20.315 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:46:20.316 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:46:20.316 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:46:20.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:46:20.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:46:20.587 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:46:20.816 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:46:20.817 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:46:20.857 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:46:20.887 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:46:20.923 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T03:46:20.924 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:46:20.963 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:46:22.899 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:48:22.963 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:48:23.002 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:48:24.922 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:48:24.940 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:48:24.940 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:48:24.941 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:48:24.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:48:25.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:48:25.200 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:48:25.444 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:48:25.444 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:48:25.481 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:48:25.514 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:48:25.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T03:48:25.547 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:48:25.590 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:48:27.409 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:50:27.465 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:50:27.507 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:50:29.388 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:50:29.406 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:50:29.406 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:50:29.406 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:50:29.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:50:29.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:50:29.666 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:50:29.899 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:50:29.899 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:50:29.938 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:50:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:50:30.047 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T03:50:30.049 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:50:30.091 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:50:31.266 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:52:31.331 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:52:31.372 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:52:33.172 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:52:33.189 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:52:33.189 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:52:33.189 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:52:33.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:52:33.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:52:33.460 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:52:33.689 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:52:33.689 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:52:33.734 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:52:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:52:33.799 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T03:52:33.801 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:52:33.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:52:34.941 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:54:35.003 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:54:35.041 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:54:37.059 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:54:37.076 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:54:37.076 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:54:37.076 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:54:37.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:54:37.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:54:37.333 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:54:37.566 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:54:37.566 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:54:37.604 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:54:37.636 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:54:37.670 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T03:54:37.672 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:54:37.715 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:54:38.875 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:56:38.935 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:56:38.977 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:56:41.111 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:56:41.128 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:56:41.128 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:56:41.128 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:56:41.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:56:41.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:56:41.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:56:41.622 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:56:41.622 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:56:41.662 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:56:41.691 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:56:41.728 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T03:56:41.729 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:56:41.769 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:56:43.313 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:58:43.369 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T03:58:43.409 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:58:45.299 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T03:58:45.315 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T03:58:45.316 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T03:58:45.316 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:58:45.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:58:45.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T03:58:45.585 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T03:58:45.806 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T03:58:45.806 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T03:58:45.847 INFO:teuthology.misc:Compressing logs... 2021-03-22T03:58:45.876 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T03:58:45.911 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T03:58:45.912 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T03:58:45.952 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T03:58:47.014 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:00:47.072 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:00:47.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:00:49.396 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:00:49.416 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:00:49.416 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:00:49.416 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:00:49.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:00:49.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:00:49.680 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:00:53.492 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:00:53.492 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:00:53.653 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:00:53.687 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:00:53.711 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:00:53.713 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:00:53.756 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:00:55.779 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:02:55.838 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:02:55.883 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:02:57.917 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:02:57.954 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:02:57.954 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:02:57.959 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:02:57.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:02:58.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:02:58.218 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:02:59.077 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:02:59.078 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:02:59.186 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:02:59.218 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:02:59.228 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:02:59.230 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:02:59.271 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:03:00.990 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:05:01.048 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:05:01.091 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:05:02.939 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:05:03.034 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:05:03.034 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:05:03.035 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:05:03.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:05:03.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:05:03.301 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:05:04.127 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:05:04.127 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:05:04.240 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:05:04.269 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:05:04.277 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:05:04.280 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:05:04.322 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:05:05.444 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:07:05.501 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:07:05.541 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:07:06.580 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:07:06.598 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:07:06.599 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:07:06.599 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:07:06.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:07:06.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:07:06.854 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:07:07.844 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:07:07.845 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:07:07.976 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:07:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:07:08.027 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:07:08.029 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:07:08.067 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:07:09.075 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:09:09.131 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:09:09.170 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:09:10.284 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:09:10.307 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:09:10.307 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:09:10.307 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:09:10.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:09:10.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:09:10.565 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:09:12.071 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:09:12.071 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:09:12.231 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:09:12.261 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:09:12.272 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:09:12.274 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:09:12.312 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:09:13.396 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:11:13.458 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:11:13.499 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:11:14.596 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:11:14.617 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:11:14.617 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:11:14.617 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:11:14.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:11:14.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:11:14.887 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:11:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:11:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:11:16.675 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:11:16.707 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:11:16.716 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:11:16.718 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:11:16.763 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:11:17.822 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:13:17.880 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:13:17.916 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:13:19.854 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:13:19.871 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:13:19.871 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:13:19.871 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:13:19.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:13:20.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:13:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:13:20.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:13:20.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:13:20.488 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:13:20.528 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:13:20.530 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:13:20.531 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:13:20.570 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:13:22.103 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:15:22.160 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:15:22.200 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:15:23.969 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:15:23.986 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:15:23.987 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:15:23.987 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:15:23.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:15:24.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:15:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:15:24.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:15:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:15:24.530 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:15:24.569 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:15:24.594 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:15:24.596 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:15:24.636 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:15:25.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:17:25.760 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:21:08.848 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:21:10.771 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:21:10.819 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:21:10.819 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:21:10.819 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:21:10.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:21:11.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:21:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:21:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:21:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:21:11.551 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:21:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:21:11.585 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:21:11.586 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:21:11.645 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:21:12.786 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:23:12.849 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:23:12.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:23:14.655 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:23:14.673 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:23:14.673 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:23:14.673 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:23:14.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:23:14.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:23:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:23:15.264 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:23:15.264 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:23:15.299 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:23:15.335 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:23:15.363 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:23:15.364 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:23:15.403 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:23:16.478 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:25:16.536 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:25:16.576 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:25:18.432 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:25:18.449 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:25:18.449 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:25:18.449 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:25:18.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:25:18.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:25:18.703 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:25:18.941 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:25:18.941 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:25:18.984 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:25:19.014 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:25:19.049 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:25:19.050 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:25:19.093 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:25:20.188 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:27:20.257 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:27:20.296 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:27:21.413 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:27:21.434 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:27:21.435 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:27:21.435 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:27:21.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:27:21.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:27:21.937 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:27:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:27:21.965 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:27:22.004 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:27:22.029 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:27:22.031 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:27:22.067 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:27:23.162 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:29:23.222 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:29:23.260 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:29:24.322 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:29:24.340 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:29:24.340 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:29:24.340 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:29:24.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:29:24.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:29:24.594 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:29:24.820 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:29:24.820 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:29:24.858 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:29:24.892 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:29:24.923 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:29:24.924 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:29:24.965 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:29:26.078 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:31:26.137 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:31:26.176 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:31:27.254 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:31:27.272 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:31:27.272 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:31:27.272 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:31:27.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:31:27.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:31:27.525 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:31:27.748 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:31:27.749 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:31:27.786 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:31:27.822 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:31:27.851 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:31:27.852 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:31:27.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:31:28.964 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:33:29.026 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:33:29.080 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:33:30.195 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:33:30.215 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:33:30.215 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:33:30.215 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:33:30.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:33:30.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:33:30.485 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:33:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:33:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:33:30.755 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:33:30.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:33:30.819 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:33:30.821 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:33:30.859 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:33:31.950 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:35:32.005 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:35:32.047 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:35:33.087 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:35:33.104 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:35:33.105 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:35:33.105 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:35:33.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:35:33.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:35:33.361 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:35:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:35:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:35:33.643 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:35:33.678 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:35:33.709 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:35:33.711 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:35:33.750 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:35:34.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:37:34.868 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:37:34.916 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:37:36.943 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:37:36.962 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:37:36.963 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:37:36.963 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:37:36.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:37:37.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:37:37.218 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:37:37.453 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:37:37.453 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:37:37.494 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:37:37.523 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:37:37.561 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:37:37.563 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:37:37.604 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:37:38.745 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:39:38.808 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:39:38.848 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:39:40.751 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:39:40.780 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:39:40.781 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:39:40.781 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:39:40.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:39:41.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:39:41.048 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:39:41.286 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:39:41.286 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:39:41.321 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:39:41.547 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:39:41.548 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:39:41.550 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:39:41.593 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:39:42.872 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:41:42.928 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:41:42.974 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:41:44.910 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:41:44.928 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:41:44.928 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:41:44.929 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:41:44.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:41:45.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:41:45.190 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:41:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:41:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:41:45.464 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:41:45.495 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:41:45.530 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:41:45.532 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:41:45.587 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:41:46.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:43:46.997 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:43:47.091 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:43:48.956 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:43:48.973 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:43:48.973 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:43:48.974 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:43:48.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:43:49.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:43:49.227 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:43:49.451 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:43:49.452 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:43:49.490 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:43:49.521 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:43:49.555 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:43:49.557 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:43:49.596 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:43:50.652 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:45:50.710 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:45:50.748 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:45:52.602 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:45:52.619 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:45:52.619 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:45:52.619 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:45:52.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:45:52.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:45:52.890 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:45:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:45:53.138 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:45:53.175 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:45:53.206 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:45:53.239 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T04:45:53.241 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T04:45:53.280 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:45:54.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:47:54.811 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:47:54.852 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:47:56.853 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:47:56.872 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:47:56.872 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:47:56.872 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:47:56.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:47:57.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:47:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:48:31.682 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T04:48:31.728 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:48:33.577 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:50:33.637 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:50:33.677 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:50:35.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:50:35.896 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:50:35.896 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:50:35.896 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:50:35.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:50:36.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:50:36.158 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:50:38.338 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T04:50:38.381 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:50:40.211 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:52:40.273 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:52:40.311 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:52:41.989 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:52:42.007 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:52:42.007 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:52:42.007 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:52:42.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:52:42.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:52:42.277 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:52:42.289 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T04:52:43.385 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T04:52:44.450 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T04:52:45.555 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T04:52:46.613 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T04:52:47.714 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T04:52:48.774 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T04:52:49.871 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T04:52:50.930 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T04:52:52.027 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T04:52:52.027 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T04:52:52.067 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:52:53.159 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:54:53.239 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:54:53.278 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:54:54.365 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:54:54.382 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:54:54.382 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:54:54.382 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:54:54.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:54:54.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:54:54.637 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:55:29.218 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T04:55:29.269 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:55:30.337 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:57:30.397 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:57:30.437 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:57:31.486 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:57:31.502 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:57:31.502 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:57:31.502 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:57:31.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:57:31.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:57:31.756 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:57:34.786 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T04:57:34.824 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:57:35.951 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:59:36.011 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T04:59:36.049 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:59:37.144 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T04:59:37.163 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T04:59:37.163 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T04:59:37.163 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:59:37.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:59:37.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T04:59:37.436 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T04:59:37.712 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T04:59:37.743 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:59:37.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:59:37.769 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T04:59:37.769 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:59:37.772 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:59:37.815 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T04:59:37.888 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T04:59:37.888 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:59:37.921 INFO:teuthology.misc:Compressing logs... 2021-03-22T04:59:37.959 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T04:59:37.963 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T04:59:38.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T04:59:38.030 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:59:38.044 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T04:59:38.045 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T04:59:38.086 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T04:59:39.144 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:01:39.204 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:01:39.247 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:01:40.304 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:01:40.320 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:01:40.320 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:01:40.320 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:01:40.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:01:40.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:01:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:01:40.871 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T05:01:40.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:01:40.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:01:40.931 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:01:40.931 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:01:40.935 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:01:40.975 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:01:41.046 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T05:01:41.046 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:01:41.084 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:01:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T05:01:41.146 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T05:01:41.191 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T05:01:41.192 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:01:41.216 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T05:01:41.216 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:01:41.265 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:01:42.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:03:42.474 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:03:42.514 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:03:43.629 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:03:43.646 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:03:43.646 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:03:43.647 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:03:43.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:03:43.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:03:43.901 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:03:44.213 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:03:44.214 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:03:44.241 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:03:44.284 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:03:44.356 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T05:03:44.357 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:03:44.398 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:03:44.428 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T05:03:44.504 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T05:03:44.504 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:03:44.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:03:45.648 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:05:45.707 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:05:45.746 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:05:46.820 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:05:46.837 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:05:46.837 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:05:46.837 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:05:46.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:05:47.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:05:47.104 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:06:47.063 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T05:06:47.101 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:06:48.185 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:08:48.243 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:08:48.277 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:08:49.243 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:08:49.261 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:08:49.261 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:08:49.261 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:08:49.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:08:49.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:08:49.518 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:08:59.714 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T05:08:59.749 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:09:00.697 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:11:00.753 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:11:00.794 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:11:01.811 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:11:01.827 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:11:01.827 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:11:01.828 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:11:01.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:11:02.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:11:02.096 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:11:02.346 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:11:02.380 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:11:02.380 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:11:02.446 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:11:02.446 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:11:02.448 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:11:02.477 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:11:02.478 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:11:02.547 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T05:11:02.547 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:11:02.572 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T05:11:02.572 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:11:02.573 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:11:02.616 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T05:11:02.636 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T05:11:02.685 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T05:11:02.685 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:11:02.709 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T05:11:02.709 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:11:02.747 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:11:03.758 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:13:03.819 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:13:03.855 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:13:04.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:13:04.868 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:13:04.868 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:13:04.869 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:13:04.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:13:05.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:13:05.120 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:13:05.316 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:13:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:13:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:13:05.383 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:13:05.383 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:13:05.384 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:13:05.419 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:13:05.448 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:13:05.489 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T05:13:05.490 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:13:05.516 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T05:13:05.516 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:13:05.517 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:13:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T05:13:05.583 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T05:13:05.628 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T05:13:05.628 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:13:05.653 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T05:13:05.653 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:13:05.689 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:13:06.621 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:15:06.681 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:15:06.720 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:15:07.797 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:15:07.814 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:15:07.814 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:15:07.815 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:15:07.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:15:08.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:15:08.073 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:16:08.045 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T05:16:08.087 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:16:09.171 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:18:09.231 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:18:09.273 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:18:10.350 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:18:10.367 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:18:10.368 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:18:10.368 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:18:10.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:18:10.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:18:10.646 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:18:10.837 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:18:10.864 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:18:10.864 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:18:10.910 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:18:10.911 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:18:10.912 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:18:10.938 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:18:11.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T05:18:11.012 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:18:11.069 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:18:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T05:18:11.115 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T05:18:11.161 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T05:18:11.161 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:18:11.187 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T05:18:11.188 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:18:11.226 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:18:12.271 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:20:12.336 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:20:12.376 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:20:13.486 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:20:13.503 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:20:13.503 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:20:13.503 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:20:13.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:20:13.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:20:13.754 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:20:13.948 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:20:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:20:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:20:14.016 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:20:14.016 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:20:14.017 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:20:14.048 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:20:14.082 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:20:14.083 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:20:14.184 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:20:15.235 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:22:15.294 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:22:15.333 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:22:16.329 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:22:16.346 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:22:16.346 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:22:16.346 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:22:16.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:22:16.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:22:16.599 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:22:16.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:22:16.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:22:16.869 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:22:16.897 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:22:16.934 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:22:16.936 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:22:16.970 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:22:17.956 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:24:18.015 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:24:18.056 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:24:19.159 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:24:19.177 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:24:19.177 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:24:19.177 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:24:19.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:24:19.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:24:19.448 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:24:19.657 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:24:19.658 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:24:19.697 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:24:19.728 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:24:19.762 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:24:19.763 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:24:19.804 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:24:20.911 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:26:20.972 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:26:21.010 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:26:22.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:26:22.057 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:26:22.057 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:26:22.057 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:26:22.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:26:22.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:26:22.315 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:26:22.553 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:26:22.553 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:26:22.591 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:26:22.623 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:26:22.661 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:26:22.663 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:26:22.700 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:26:23.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:28:23.758 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:28:23.795 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:28:24.784 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:28:24.801 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:28:24.801 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:28:24.802 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:28:24.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:28:25.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:28:25.069 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:28:25.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:28:25.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:28:25.357 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:28:25.397 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:28:25.425 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:28:25.425 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:28:25.462 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:28:26.480 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:30:26.539 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:30:26.578 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:30:27.564 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:30:27.583 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:30:27.583 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:30:27.583 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:30:27.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:30:27.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:30:27.855 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:30:28.063 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:30:28.063 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:30:28.102 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:30:28.133 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:30:28.166 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:30:28.168 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:30:28.205 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:30:29.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:32:29.240 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:32:29.278 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:32:30.311 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:32:30.329 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:32:30.329 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:32:30.329 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:32:30.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:32:30.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:32:30.591 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:32:30.831 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:32:30.989 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:32:31.040 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:32:31.076 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:32:31.108 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:32:31.110 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:32:31.150 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:32:32.254 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:34:32.313 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:34:32.355 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:34:33.441 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:34:33.458 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:34:33.458 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:34:33.458 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:34:33.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:34:33.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:34:33.714 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:34:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:34:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:34:34.003 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:34:34.033 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:34:34.070 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:34:34.073 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:34:34.112 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:34:35.167 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:36:35.239 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:36:35.276 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:36:36.252 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:36:36.268 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:36:36.268 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:36:36.268 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:36:36.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:36:36.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:36:36.534 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:36:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:36:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:36:36.787 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:36:36.817 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:36:36.855 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:36:36.856 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:36:36.894 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:36:37.883 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:38:37.944 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:38:37.980 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:38:38.980 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:38:38.996 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:38:38.996 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:38:38.996 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:38:38.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:38:39.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:38:39.249 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:38:39.483 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:38:39.483 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:38:39.523 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:38:39.553 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:38:39.589 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:38:39.590 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:38:39.627 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:38:40.624 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:40:40.681 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:40:40.722 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:40:41.725 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:40:41.741 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:40:41.741 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:40:41.741 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:40:41.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:40:41.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:40:41.993 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:40:42.254 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:40:42.255 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:40:42.286 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:40:42.326 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:40:42.352 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:40:42.354 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:40:42.389 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:40:43.370 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:42:43.426 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:42:43.468 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:42:44.608 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:42:44.625 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:42:44.625 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:42:44.625 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:42:44.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:42:44.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:42:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:42:45.101 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:42:45.102 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:42:45.142 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:42:45.171 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:42:45.207 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:42:45.208 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:42:45.245 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:42:46.273 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:44:46.334 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:44:46.368 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:44:47.365 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:44:47.381 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:44:47.382 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:44:47.382 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:44:47.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:44:47.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:44:47.636 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:44:47.871 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:44:47.871 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:44:47.913 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:44:47.940 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:44:47.981 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:44:47.983 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:44:48.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:44:49.037 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:46:49.098 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:46:49.135 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:46:50.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:46:50.214 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:46:50.214 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:46:50.214 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:46:50.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:46:50.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:46:50.467 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:46:50.708 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:46:50.708 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:46:50.745 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:46:50.778 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:46:50.811 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:46:50.812 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:46:50.852 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:46:51.872 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:48:51.930 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:48:51.972 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:48:53.063 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:48:53.080 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:48:53.080 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:48:53.080 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:48:53.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:48:53.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:48:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:48:53.565 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:48:53.565 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:48:53.604 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:48:53.635 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:48:53.670 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:48:53.674 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:48:53.714 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:48:54.742 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:50:54.802 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:50:54.839 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:50:55.831 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:50:55.847 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:50:55.847 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:50:55.848 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:50:55.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:50:56.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:50:56.100 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:50:56.336 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:50:56.336 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:50:56.375 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:50:56.405 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:50:56.440 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:50:56.442 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:50:56.481 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:50:57.491 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:52:57.547 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:52:57.586 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:52:58.630 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:52:58.647 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:52:58.647 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:52:58.648 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:52:58.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:52:58.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:52:58.901 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:52:59.165 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:52:59.166 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:52:59.200 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:52:59.238 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:52:59.267 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:52:59.269 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:52:59.307 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:53:00.361 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:55:00.436 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:55:00.475 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:55:01.507 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:55:01.578 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:55:01.579 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:55:01.579 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:55:01.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:55:01.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:55:01.842 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:55:02.052 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:55:02.052 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:55:02.092 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:55:02.123 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:55:02.157 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:55:02.158 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:55:02.196 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:55:03.173 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:57:03.226 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:57:03.261 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:57:04.332 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:57:04.348 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:57:04.348 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:57:04.348 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:57:04.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:57:04.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:57:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:57:04.829 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:57:04.829 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:57:04.866 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:57:04.899 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:57:04.931 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:57:04.933 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:57:04.971 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:57:05.967 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:59:06.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T05:59:06.065 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:59:07.101 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T05:59:07.118 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T05:59:07.119 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T05:59:07.119 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:59:07.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:59:07.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T05:59:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T05:59:07.634 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T05:59:07.634 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T05:59:07.665 INFO:teuthology.misc:Compressing logs... 2021-03-22T05:59:07.702 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T05:59:07.730 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T05:59:07.731 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T05:59:07.770 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T05:59:08.805 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:01:08.865 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:01:08.903 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:01:09.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:01:09.989 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:01:09.989 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:01:09.989 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:01:09.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:01:10.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:01:10.262 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:01:10.484 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:01:10.484 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:01:10.523 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:01:10.555 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:01:10.591 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:01:10.592 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:01:10.629 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:01:11.629 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:03:11.687 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:03:11.724 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:03:12.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:03:12.705 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:03:12.705 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:03:12.705 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:03:12.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:03:12.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:03:12.955 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:03:13.180 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:03:13.180 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:03:13.220 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:03:13.250 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:03:13.287 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:03:13.288 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:03:13.322 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:03:14.260 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:05:14.316 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:05:14.353 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:05:15.416 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:05:15.433 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:05:15.433 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:05:15.434 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:05:15.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:05:15.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:05:15.688 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:05:15.926 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:05:15.927 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:05:15.964 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:05:15.996 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:05:16.028 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:05:16.030 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:05:16.067 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:05:17.114 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:07:17.171 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:07:17.209 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:07:18.281 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:07:18.303 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:07:18.303 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:07:18.303 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:07:18.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:07:18.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:07:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:07:18.782 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:07:18.782 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:07:18.821 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:07:18.852 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:07:18.886 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:07:18.888 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:07:18.924 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:07:19.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:09:20.054 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:09:20.094 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:09:21.143 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:09:21.160 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:09:21.160 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:09:21.161 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:09:21.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:09:21.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:09:21.416 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:09:21.650 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:09:21.650 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:09:21.690 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:09:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:09:21.755 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:09:21.757 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:09:21.797 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:09:22.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:11:22.903 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:11:22.942 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:11:23.964 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:11:23.982 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:11:23.982 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:11:23.982 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:11:23.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:11:24.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:11:24.234 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:11:24.483 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:11:24.483 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:11:24.511 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:11:24.553 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:11:24.577 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:11:24.578 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:11:24.615 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:11:25.627 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:13:25.685 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:13:25.741 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:13:26.833 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:13:26.853 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:13:26.853 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:13:26.853 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:13:26.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:13:27.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:13:27.126 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:13:27.339 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:13:27.340 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:13:27.380 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:13:27.408 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:13:27.446 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:13:27.448 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:13:27.487 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:13:28.530 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:15:28.669 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:15:28.707 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:15:29.742 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:15:29.759 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:15:29.760 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:15:29.760 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:15:29.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:15:29.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:15:30.022 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:15:30.250 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:15:30.250 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:15:30.287 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:15:30.322 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:15:30.353 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:15:30.355 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:15:30.396 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:15:31.484 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:17:31.542 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:17:31.580 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:17:32.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:17:32.659 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:17:32.659 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:17:32.659 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:17:32.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:17:32.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:17:32.912 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:17:33.159 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:17:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:17:33.198 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:17:33.230 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:17:33.264 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:17:33.266 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:17:33.303 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:17:34.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:19:34.409 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:19:34.448 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:19:35.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:19:35.513 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:19:35.514 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:19:35.514 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:19:35.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:19:35.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:19:35.782 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:19:35.996 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:19:35.997 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:19:36.034 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:19:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:19:36.099 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:19:36.101 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:19:36.138 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:19:37.166 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:21:37.232 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:21:37.272 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:21:38.329 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:21:38.345 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:21:38.346 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:21:38.346 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:21:38.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:21:38.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:21:38.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:21:38.832 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:21:38.832 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:21:38.872 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:21:38.902 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:21:38.938 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:21:38.939 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:21:38.979 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:21:40.026 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:23:40.085 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:23:40.124 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:23:41.166 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:23:41.184 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:23:41.185 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:23:41.185 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:23:41.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:23:41.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:23:41.440 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:23:41.688 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:23:41.688 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:23:41.716 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:23:41.759 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:23:41.783 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:23:41.784 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:23:41.821 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:23:42.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:25:42.874 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:25:42.917 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:25:44.057 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:25:44.079 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:25:44.079 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:25:44.079 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:25:44.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:25:44.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:25:44.349 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:25:44.562 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:25:44.563 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:25:44.606 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:25:44.636 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:25:44.671 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:25:44.673 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:25:44.712 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:25:45.878 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:27:45.934 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:27:45.974 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:27:47.041 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:27:47.060 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:27:47.060 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:27:47.060 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:27:47.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:27:47.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:27:47.313 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:27:47.547 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:27:47.547 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:27:47.586 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:27:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:27:47.653 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:27:47.654 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:27:47.695 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:27:48.781 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:29:48.841 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:29:48.949 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:29:50.079 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:29:50.099 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:29:50.099 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:29:50.099 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:29:50.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:29:50.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:29:50.362 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:29:50.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:29:50.609 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:29:50.640 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:29:50.679 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:29:50.705 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:29:50.706 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:29:50.750 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:29:51.891 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:31:51.977 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:31:52.017 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:31:53.099 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:31:53.117 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:31:53.117 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:31:53.117 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:31:53.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:31:53.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:31:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:31:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:31:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:31:53.649 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:31:53.675 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:31:53.714 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:31:53.716 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:31:53.756 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:31:54.866 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:33:54.943 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:34:45.996 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:34:47.049 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:34:47.066 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:34:47.067 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:34:47.067 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:34:47.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:34:47.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:34:47.329 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:34:47.575 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:34:47.575 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:34:47.614 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:34:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:34:47.683 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:34:47.683 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:34:47.730 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:34:48.827 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:36:48.887 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:36:48.929 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:36:50.107 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:36:50.127 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:36:50.128 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:36:50.128 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:36:50.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:36:50.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:36:50.389 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:36:50.650 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:36:50.650 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:36:50.684 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:36:50.722 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:36:50.750 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:36:50.752 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:36:50.792 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:36:51.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:38:52.001 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:38:52.044 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:38:54.374 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:38:54.411 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:38:54.411 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:38:54.411 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:38:54.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:38:54.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:38:54.699 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:39:13.094 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T06:39:13.144 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:39:14.991 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:41:15.055 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:41:15.098 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:41:17.194 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:41:17.213 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:41:17.214 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:41:17.214 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:41:17.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:41:17.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:41:17.469 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:41:17.499 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T06:41:18.555 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T06:41:19.671 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T06:41:20.773 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T06:41:21.878 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T06:41:22.986 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T06:41:24.097 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T06:41:25.196 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T06:41:26.255 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T06:41:27.351 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T06:41:27.352 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T06:41:27.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:41:28.520 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:43:28.581 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:43:28.621 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:43:30.519 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:43:30.541 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:43:30.542 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:43:30.542 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:43:30.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:43:30.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:43:30.809 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:43:47.264 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:43:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:43:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:43:47.331 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:43:47.331 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:43:47.333 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:43:47.366 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:43:47.396 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:43:47.438 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T06:43:47.438 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:43:47.463 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T06:43:47.463 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:43:47.465 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:43:47.507 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T06:43:47.531 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T06:43:47.576 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T06:43:47.577 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:43:47.603 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T06:43:47.603 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:43:47.647 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:43:48.756 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:45:48.824 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:45:48.875 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:45:50.055 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:45:50.074 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:45:50.074 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:45:50.074 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:45:50.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:45:50.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:45:50.349 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:45:50.528 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:45:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:45:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:45:50.597 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:45:50.598 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:45:50.599 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:45:50.632 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:45:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T06:45:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:45:50.760 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:45:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T06:45:50.809 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T06:45:50.858 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T06:45:50.858 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:45:50.881 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T06:45:50.882 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:45:50.923 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:45:52.085 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:47:52.146 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:47:52.186 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:47:53.284 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:47:53.302 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:47:53.302 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:47:53.303 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:47:53.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:47:53.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:47:53.563 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:47:53.769 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:47:53.800 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:47:53.800 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:47:53.835 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:47:53.835 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:47:53.837 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:47:53.870 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:47:53.902 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:47:53.903 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:47:53.948 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:47:55.151 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:49:55.210 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:49:55.249 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:49:56.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:49:56.353 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:49:56.353 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:49:56.353 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:49:56.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:49:56.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:49:56.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:49:56.801 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:49:56.830 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:49:56.831 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:49:56.867 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:49:56.867 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:49:56.868 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:49:56.906 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:49:56.933 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:49:56.935 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:49:56.978 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:49:57.985 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:51:58.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:51:58.093 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:51:59.138 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:51:59.155 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:51:59.155 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:51:59.155 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:51:59.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:51:59.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:51:59.426 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:51:59.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:51:59.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:51:59.674 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:51:59.706 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:51:59.738 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:51:59.740 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:51:59.777 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:52:00.843 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:54:00.902 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:54:00.940 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:54:02.051 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:54:02.066 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:54:02.067 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:54:02.067 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:54:02.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:54:02.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:54:02.324 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:54:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:54:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:54:02.610 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:54:02.638 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:54:02.694 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:54:02.695 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:54:02.732 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:54:03.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:56:03.892 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:56:03.928 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:56:04.924 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:56:04.944 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:56:04.944 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:56:04.944 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:56:04.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:56:05.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:56:05.200 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:56:05.432 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:56:05.432 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:56:05.469 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:56:05.500 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:56:05.533 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:56:05.535 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:56:05.570 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:56:06.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:58:06.596 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T06:58:06.637 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:58:07.656 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T06:58:07.672 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T06:58:07.672 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T06:58:07.672 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:58:07.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:58:07.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T06:58:07.940 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T06:58:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T06:58:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T06:58:08.207 INFO:teuthology.misc:Compressing logs... 2021-03-22T06:58:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T06:58:08.273 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T06:58:08.274 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T06:58:08.313 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T06:58:09.266 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:00:09.332 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:00:09.374 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:00:10.510 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:00:10.528 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:00:10.528 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:00:10.529 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:00:10.529 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:00:10.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:00:10.784 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:00:11.028 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:00:11.028 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:00:11.067 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:00:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:00:11.135 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T07:00:11.137 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T07:00:11.193 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:00:12.357 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:02:12.414 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:02:12.452 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:02:13.480 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:02:13.495 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:02:13.496 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:02:13.496 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:02:13.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:02:13.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:02:13.754 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:02:32.130 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T07:02:32.169 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:02:33.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:04:33.196 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:04:33.232 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:04:34.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:04:34.374 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:04:34.374 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:04:34.374 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:04:34.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:04:34.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:04:34.640 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:04:34.658 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:04:35.754 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:04:36.911 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:04:38.008 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:04:39.111 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:04:40.211 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:04:41.272 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:04:42.331 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:04:43.430 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:04:44.527 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:04:44.527 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T07:04:44.563 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:04:45.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:06:45.646 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:06:45.685 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:06:46.716 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:06:46.732 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:06:46.732 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:06:46.732 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:06:46.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:06:46.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:06:46.988 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:07:46.962 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T07:07:47.002 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:07:48.035 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:09:48.094 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:09:48.131 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:09:49.172 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:09:49.188 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:09:49.188 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:09:49.188 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:09:49.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:09:49.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:09:49.461 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:09:49.917 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T07:09:49.949 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:09:49.949 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:09:49.976 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:09:49.976 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:09:49.977 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:09:50.020 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:09:50.034 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T07:09:50.090 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:09:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:09:50.108 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:09:50.108 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:09:50.111 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:09:50.160 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:09:50.195 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T07:09:50.228 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:09:50.229 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:09:50.251 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:09:50.251 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:09:50.291 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:09:51.266 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:11:51.327 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:11:51.366 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:11:52.374 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:11:52.390 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:11:52.390 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:11:52.390 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:11:52.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:11:52.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:11:52.644 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:11:52.753 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T07:11:52.793 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:11:53.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:13:53.869 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:13:53.908 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:13:54.973 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:13:54.990 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:13:54.990 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:13:54.990 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:13:54.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:13:55.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:13:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:13:55.573 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T07:13:55.605 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:13:55.605 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:13:55.631 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:13:55.632 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:13:55.633 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:13:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:13:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:13:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:13:55.781 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:13:55.825 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:13:55.839 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T07:13:55.894 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:13:55.894 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:13:55.910 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:13:55.910 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:13:55.951 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:13:56.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:15:57.063 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:15:57.100 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:15:58.107 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:15:58.123 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:15:58.123 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:15:58.123 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:15:58.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:15:58.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:15:58.393 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:15:58.690 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:15:58.691 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:15:58.721 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:15:58.761 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:15:58.832 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:15:58.832 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:15:58.872 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:15:58.902 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:15:58.973 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:15:58.973 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:15:59.030 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:16:00.000 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:18:00.069 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:18:00.106 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:18:01.214 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:18:01.232 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:18:01.232 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:18:01.232 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:18:01.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:18:01.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:18:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:18:01.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:18:01.822 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:18:01.854 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:18:01.892 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:18:01.963 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:18:01.963 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:18:02.002 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:18:02.031 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:18:02.100 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:18:02.100 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:18:02.156 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:18:03.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:20:03.251 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:20:03.290 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:20:04.345 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:20:04.377 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:20:04.377 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:20:04.377 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:20:04.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:20:04.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:20:04.633 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:20:04.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:20:04.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:20:05.007 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:20:05.043 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:20:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:20:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:20:05.154 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:20:05.182 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:20:05.251 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:20:05.252 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:20:05.310 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:20:06.350 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:22:06.416 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:22:06.456 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:22:07.544 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:22:07.560 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:22:07.561 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:22:07.561 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:22:07.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:22:07.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:22:07.831 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:22:08.442 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:22:08.442 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:22:08.472 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:22:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:22:08.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:22:08.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:22:08.645 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:22:08.673 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:22:08.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:22:08.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:22:08.815 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:22:09.882 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:24:09.944 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:24:09.984 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:24:11.035 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:24:11.052 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:24:11.052 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:24:11.052 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:24:11.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:24:11.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:24:11.303 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:24:29.762 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T07:24:29.798 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:24:30.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:26:30.893 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:26:30.929 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:26:31.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:26:32.008 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:26:32.008 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:26:32.008 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:26:32.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:26:32.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:26:32.261 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:26:32.290 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:26:33.352 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:26:34.412 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:26:35.509 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:26:36.605 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:26:37.703 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:26:38.799 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:26:39.896 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:26:40.997 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:26:42.098 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T07:26:42.098 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T07:26:42.134 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:26:43.119 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:28:43.179 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:28:43.221 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:28:44.359 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:28:44.376 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:28:44.376 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:28:44.376 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:28:44.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:28:44.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:28:44.654 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:29:03.171 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T07:29:03.212 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:29:04.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:31:04.291 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:31:04.329 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:31:05.369 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:31:05.385 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:31:05.385 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:31:05.385 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:31:05.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:31:05.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:31:05.642 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:31:05.943 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T07:31:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:31:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:31:06.002 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:31:06.002 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:31:06.005 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:31:06.046 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:31:06.060 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T07:31:06.116 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:31:06.116 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:31:06.132 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:31:06.132 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:31:06.134 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:31:06.185 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:31:06.189 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T07:31:06.254 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:31:06.254 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:31:06.269 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:31:06.269 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:31:06.308 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:31:07.409 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:33:07.472 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:33:07.508 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:33:08.538 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:33:08.558 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:33:08.558 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:33:08.559 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:33:08.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:33:08.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:33:08.813 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:33:09.205 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T07:33:09.237 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:33:09.237 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:33:09.253 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:33:09.253 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:33:09.255 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:33:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:33:09.312 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T07:33:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:33:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:33:09.396 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:33:09.396 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:33:09.398 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:33:09.447 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:33:09.558 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T07:33:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:33:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:33:09.601 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:33:09.601 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:33:09.641 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:33:10.744 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:35:10.804 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:35:10.844 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:35:11.978 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:35:11.998 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:35:11.999 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:35:11.999 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:35:11.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:35:12.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:35:12.278 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:35:12.566 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T07:35:12.598 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:35:12.598 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:35:12.628 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:35:12.628 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:35:12.631 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:35:12.668 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:35:12.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:35:12.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:35:12.773 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:35:12.809 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:35:12.813 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T07:35:12.878 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:35:12.878 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:35:12.894 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:35:12.894 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:35:12.935 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:35:13.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:37:14.039 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:37:14.085 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:37:15.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:37:15.301 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:37:15.301 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:37:15.301 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:37:15.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:37:15.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:37:15.652 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:37:16.072 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T07:37:16.123 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:37:16.123 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:37:16.146 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:37:16.147 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:37:16.148 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:37:16.185 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:37:16.262 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:37:16.263 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:37:16.337 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:37:16.369 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T07:37:16.379 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:37:16.464 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:37:16.465 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:37:16.483 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:37:16.483 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:37:16.522 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:37:17.576 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:39:17.639 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:39:17.678 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:39:18.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:39:18.714 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:39:18.714 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:39:18.714 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:39:18.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:39:18.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:39:18.968 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:39:19.310 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:39:19.311 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:39:19.335 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:39:19.380 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:39:19.451 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:39:19.451 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:39:19.488 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:39:19.520 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:39:19.589 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:39:19.589 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:39:19.702 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:39:20.697 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:41:20.759 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:41:20.797 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:41:21.891 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:41:21.908 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:41:21.908 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:41:21.909 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:41:21.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:41:22.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:41:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:41:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:41:22.536 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:41:22.574 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:41:22.644 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:41:22.644 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:41:22.685 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:41:22.712 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:41:22.782 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T07:41:22.782 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:41:22.882 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:41:23.961 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:43:24.023 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:43:24.062 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:43:25.274 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:43:25.291 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:43:25.291 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:43:25.291 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:43:25.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:43:25.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:43:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:43:25.881 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:43:25.882 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:43:25.905 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:43:25.952 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:43:26.021 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:43:26.022 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:43:26.047 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:43:26.085 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-22T07:43:26.089 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:43:26.092 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T07:43:26.131 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:43:27.154 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:45:27.213 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:45:27.253 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:45:28.324 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:45:28.341 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:45:28.341 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:45:28.341 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:45:28.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:45:28.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:45:28.594 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:45:28.904 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:45:28.904 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:45:28.934 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:45:28.974 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:45:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:45:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:45:29.076 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:45:29.105 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-22T07:45:29.113 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:45:29.116 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T07:45:29.154 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:45:30.192 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:47:30.253 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:47:30.290 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:47:31.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:47:31.357 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:47:31.358 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:47:31.358 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:47:31.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:47:31.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:47:31.627 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:47:31.939 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:47:31.940 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:47:31.970 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:47:32.011 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:47:32.082 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:47:32.083 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:47:32.120 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:47:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:47:32.159 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-22T07:47:32.160 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T07:47:32.198 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:47:33.237 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:49:33.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:49:33.338 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:49:34.387 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:49:34.404 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:49:34.405 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:49:34.405 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:49:34.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:49:34.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:49:34.669 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:49:34.992 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:49:34.992 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:49:35.017 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:49:35.062 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:49:35.133 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:49:35.133 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:49:35.173 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:49:35.201 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:49:35.210 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-22T07:49:35.213 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T07:49:35.250 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:49:36.299 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:51:36.363 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:51:36.403 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:51:37.465 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:51:37.482 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:51:37.482 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:51:37.482 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:51:37.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:51:37.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:51:37.737 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:51:38.065 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:51:38.065 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:51:38.097 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:51:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:51:38.202 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:51:38.202 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:51:38.248 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:51:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:51:38.285 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-22T07:51:38.286 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T07:51:38.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:51:39.338 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:53:39.394 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:53:39.432 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:53:40.483 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:53:40.502 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:53:40.502 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:53:40.502 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:53:40.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:53:40.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:53:40.772 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:53:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:53:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:53:41.102 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:53:41.147 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:53:41.217 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:53:41.217 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:53:41.245 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:53:41.284 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:53:41.287 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-22T07:53:41.289 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T07:53:41.326 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:53:42.360 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:55:42.422 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:55:42.460 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:55:43.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:55:43.558 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:55:43.558 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:55:43.558 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:55:43.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:55:43.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:55:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:55:44.190 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:55:44.191 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:55:44.223 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:55:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:55:44.332 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:55:44.333 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:55:44.371 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:55:44.406 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T07:55:44.409 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-22T07:55:44.411 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T07:55:44.449 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:55:45.492 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:57:45.579 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:57:45.619 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:57:46.681 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:57:46.698 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:57:46.698 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:57:46.699 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:57:46.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:57:46.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:57:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:57:57.059 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T07:57:57.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:57:58.105 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:59:58.163 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T07:59:58.203 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T07:59:59.256 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T07:59:59.273 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T07:59:59.274 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T07:59:59.274 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:59:59.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:59:59.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T07:59:59.548 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T07:59:59.854 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T07:59:59.854 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T07:59:59.887 INFO:teuthology.misc:Compressing logs... 2021-03-22T07:59:59.925 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T07:59:59.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T07:59:59.994 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:00:00.046 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:00:00.072 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:00:00.088 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-22T08:00:00.090 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T08:00:00.128 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:00:01.241 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:02:01.301 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:02:01.341 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:02:02.408 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:02:02.425 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:02:02.425 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:02:02.425 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:02:02.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:02:02.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:02:02.682 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:02:03.023 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:02:03.023 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:02:03.055 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:02:03.092 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:02:03.162 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:02:03.163 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:02:03.203 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:02:03.230 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:02:03.242 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-22T08:02:03.245 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T08:02:03.282 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:02:04.296 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:04:04.356 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:04:04.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:04:05.481 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:04:05.498 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:04:05.498 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:04:05.498 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:04:05.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:04:05.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:04:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:04:06.089 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:04:06.090 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:04:06.125 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:04:06.159 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:04:06.227 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:04:06.227 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:04:06.262 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:04:06.295 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:04:06.305 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2021-03-22T08:04:06.308 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T08:04:06.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:04:07.390 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:06:07.447 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:06:07.485 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:06:08.539 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:06:08.555 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:06:08.555 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:06:08.555 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:06:08.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:06:08.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:06:08.824 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:06:27.266 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T08:06:27.304 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:06:28.319 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:08:28.380 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:08:28.430 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:08:29.481 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:08:29.497 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:08:29.498 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:08:29.498 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:08:29.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:08:29.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:08:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:08:29.826 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:08:30.929 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:08:31.989 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:09:32.992 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T08:09:33.030 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:09:34.053 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:11:34.116 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:11:34.155 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:11:35.158 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:11:35.175 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:11:35.175 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:11:35.175 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:11:35.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:11:35.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:11:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:11:35.703 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:11:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:11:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:11:35.804 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:11:35.805 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:11:35.806 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:11:35.831 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:11:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:11:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:11:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:11:35.926 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:11:35.926 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:11:35.927 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:11:35.970 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:11:35.990 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:11:36.037 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:11:36.037 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:11:36.061 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:11:36.061 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:11:36.100 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:11:37.056 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:13:37.114 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:13:37.152 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:13:38.247 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:13:38.264 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:13:38.264 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:13:38.264 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:13:38.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:13:38.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:13:38.515 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:13:38.775 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:13:38.812 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:13:38.813 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:13:38.841 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:13:38.842 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:13:38.843 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:13:38.882 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:13:38.907 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:13:38.953 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:13:38.953 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:13:38.979 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:13:38.979 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:13:38.981 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:13:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:13:39.044 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:13:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:13:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:13:39.120 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:13:39.120 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:13:39.160 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:13:40.195 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:15:40.255 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:15:40.293 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:15:41.339 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:15:41.412 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:15:41.412 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:15:41.412 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:15:41.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:15:41.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:15:41.658 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:15:41.938 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:15:41.970 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:15:41.970 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:15:42.004 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:15:42.004 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:15:42.005 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:15:42.041 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:15:42.068 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:15:42.113 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:15:42.113 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:15:42.137 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:15:42.137 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:15:42.138 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:15:42.184 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:15:42.201 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:15:42.254 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:15:42.255 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:15:42.278 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:15:42.278 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:15:42.316 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:15:43.338 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:17:43.395 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:17:43.434 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:17:44.473 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:17:44.490 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:17:44.490 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:17:44.490 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:17:44.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:17:44.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:17:44.761 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:17:45.004 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:17:45.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:17:45.043 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:17:45.071 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:17:45.071 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:17:45.072 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:17:45.113 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:17:45.137 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:17:45.185 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:17:45.185 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:17:45.209 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:17:45.209 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:17:45.210 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:17:45.254 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:17:45.273 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:17:45.323 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:17:45.323 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:17:45.347 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:17:45.347 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:17:45.386 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:17:46.434 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:19:46.496 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:19:46.538 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:19:47.612 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:19:47.630 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:19:47.630 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:19:47.630 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:19:47.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:19:47.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:19:47.883 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:19:48.145 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:19:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:19:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:19:48.211 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:19:48.212 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:19:48.213 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:19:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:19:48.278 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:19:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:19:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:19:48.352 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:19:48.352 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:19:48.353 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:19:48.396 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:19:48.416 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:19:48.467 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:19:48.467 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:19:48.491 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:19:48.491 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:19:48.531 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:19:49.652 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:21:49.715 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:21:49.754 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:21:50.908 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:21:50.930 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:21:50.931 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:21:50.931 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:21:50.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:21:51.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:21:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:21:51.426 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T08:21:51.468 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:21:52.652 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:23:52.722 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:23:52.760 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:23:53.811 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:23:53.828 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:23:53.829 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:23:53.829 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:23:53.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:23:54.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:23:54.097 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:23:54.155 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:23:55.257 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:23:56.364 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:23:57.422 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:23:58.520 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:23:59.577 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:24:00.673 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:24:01.770 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:24:02.829 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:24:03.886 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:24:03.886 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T08:24:03.926 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:24:04.985 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:26:05.044 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:26:05.084 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:26:06.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:26:06.157 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:26:06.157 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:26:06.157 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:26:06.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:26:06.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:26:06.385 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T08:26:06.425 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:26:07.445 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:28:07.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:28:07.544 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:28:08.624 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:28:08.643 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:28:08.643 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:28:08.643 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:28:08.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:28:08.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:28:08.904 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:28:09.040 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T08:28:09.071 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:28:09.071 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:28:09.095 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:28:09.096 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:28:09.097 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:28:09.129 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T08:28:09.140 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:28:09.208 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:28:09.209 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:28:09.224 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:28:09.224 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:28:09.226 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:28:09.276 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:28:09.283 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T08:28:09.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:28:09.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:28:09.362 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:28:09.362 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:28:09.400 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:28:10.473 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:30:10.535 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:30:10.585 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:30:11.729 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:30:11.746 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:30:11.746 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:30:11.747 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:30:11.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:30:11.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:30:12.014 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:30:12.159 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T08:30:12.207 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:30:12.207 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:30:12.256 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:30:12.256 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:30:12.258 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:30:12.286 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:30:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:30:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:30:12.391 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:30:12.412 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T08:30:12.419 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:30:12.486 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:30:12.486 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:30:12.503 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:30:12.504 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:30:12.559 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:30:13.587 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:32:13.651 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:32:13.691 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:32:14.785 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:32:14.804 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:32:14.804 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:32:14.805 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:32:14.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:32:15.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:32:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:32:15.212 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T08:32:15.243 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:32:15.243 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:32:15.259 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:32:15.260 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:32:15.262 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:32:15.310 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:32:15.379 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:32:15.379 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:32:15.419 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:32:15.449 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T08:32:15.453 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:32:15.523 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:32:15.523 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:32:15.545 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:32:15.545 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:32:15.586 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:32:16.594 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:34:16.654 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:34:16.696 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:34:17.772 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:34:17.790 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:34:17.791 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:34:17.791 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:34:17.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:34:18.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:34:18.043 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:34:18.195 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T08:34:18.226 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:34:18.227 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:34:18.256 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:34:18.257 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:34:18.259 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:34:18.296 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:34:18.365 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:34:18.366 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:34:18.406 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:34:18.439 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T08:34:18.445 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:34:18.516 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:34:18.516 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:34:18.539 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:34:18.539 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:34:18.579 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:34:19.744 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:36:19.805 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:36:19.845 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:36:20.936 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:36:20.953 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:36:20.953 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:36:20.953 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:36:20.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:36:21.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:36:21.224 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:36:21.367 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T08:36:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:36:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:36:21.415 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:36:21.415 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:36:21.417 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:36:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:36:21.535 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:36:21.536 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:36:21.571 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:36:21.613 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:36:21.614 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T08:36:21.679 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:36:21.680 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:36:21.702 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:36:21.702 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:36:21.744 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:36:22.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:38:22.850 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:38:22.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:38:23.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:38:23.918 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:38:23.919 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:38:23.919 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:38:23.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:38:24.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:38:24.173 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:38:24.321 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T08:38:24.353 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:38:24.354 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:38:24.368 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:38:24.368 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:38:24.370 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:38:24.423 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:38:24.493 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:38:24.493 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:38:24.532 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:38:24.566 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:38:24.638 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:38:24.638 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:38:24.710 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:38:25.766 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:40:25.829 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:40:25.868 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:40:26.884 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:40:26.900 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:40:26.901 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:40:26.901 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:40:26.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:40:27.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:40:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:40:27.325 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T08:40:27.356 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:40:27.356 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:40:27.391 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:40:27.391 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:40:27.394 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:40:27.426 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:40:27.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:40:27.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:40:27.540 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:40:27.575 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:40:27.576 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T08:40:27.614 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T08:40:27.651 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:40:28.644 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:42:28.701 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:42:28.758 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:42:29.898 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:42:29.914 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:42:29.914 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:42:29.915 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:42:29.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:42:30.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:42:30.187 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:42:30.319 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T08:42:30.349 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:42:30.349 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:42:30.368 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:42:30.368 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:42:30.371 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:42:30.419 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:42:30.487 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:42:30.488 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:42:30.530 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:42:30.562 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:42:30.571 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T08:42:30.571 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/985a3fd0-8ae9-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T08:42:30.661 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/985a3fd0-8ae9-11eb-90a1-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2021-03-22T08:42:30.748 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/985a3fd0-8ae9-11eb-90a1-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-03-22T08:42:30.961 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/985a3fd0-8ae9-11eb-90a1-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-03-22T08:42:30.963 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T08:42:31.003 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:42:32.018 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:44:32.079 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:44:32.121 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:44:33.190 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:44:33.206 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:44:33.206 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:44:33.206 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:44:33.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:44:33.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:44:33.472 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:44:33.629 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T08:44:33.662 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:44:33.662 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:44:33.678 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:44:33.678 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:44:33.680 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:44:33.732 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:44:33.802 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:44:33.802 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:44:33.841 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:44:33.871 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T08:44:33.871 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/985a3fd0-8ae9-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T08:44:33.878 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:44:33.880 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T08:44:33.918 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:44:34.969 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:46:35.028 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:46:35.070 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:46:36.139 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:46:36.156 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:46:36.156 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:46:36.156 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:46:36.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:46:36.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:46:36.414 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:46:54.919 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T08:46:54.957 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:46:56.046 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:48:56.109 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:48:56.150 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:48:57.213 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:48:57.230 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:48:57.230 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:48:57.230 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:48:57.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:48:57.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:48:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:48:57.561 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:48:58.661 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:48:59.720 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:49:00.816 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:49:01.912 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:49:03.012 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:49:04.113 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:49:05.171 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:49:06.233 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:49:07.333 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T08:49:07.333 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T08:49:07.442 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:49:08.534 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:51:08.594 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:51:08.631 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:51:09.635 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:51:09.653 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:51:09.653 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:51:09.653 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:51:09.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:51:09.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:51:09.918 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:51:10.126 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:51:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:51:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:51:10.208 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:51:10.208 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:51:10.209 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:51:10.238 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:51:10.272 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:51:10.309 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:51:10.309 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:51:10.337 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:51:10.338 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:51:10.339 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:51:10.379 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:51:10.402 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:51:10.449 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:51:10.449 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:51:10.475 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:51:10.475 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:51:10.534 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:51:11.533 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:53:11.697 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:53:11.737 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:53:12.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:53:12.791 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:53:12.792 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:53:12.792 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:53:12.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:53:13.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:53:13.058 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:53:13.264 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:53:13.294 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:53:13.294 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:53:13.330 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:53:13.330 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:53:13.331 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:53:13.363 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:53:13.433 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T08:53:13.433 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:53:13.491 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:53:13.520 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:53:13.531 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T08:53:13.590 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:53:13.590 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:53:13.616 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T08:53:13.616 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:53:13.654 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:53:14.694 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:55:14.756 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:55:14.794 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:55:15.878 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:55:15.895 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:55:15.895 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:55:15.895 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:55:15.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:55:16.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:55:16.174 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:55:16.356 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:55:16.387 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:55:16.388 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:55:16.422 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:55:16.422 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:55:16.423 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:55:16.458 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:55:16.489 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T08:55:16.490 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T08:55:16.531 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:55:17.586 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:57:17.647 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:57:17.687 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:57:18.692 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:57:18.709 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:57:18.709 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:57:18.709 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:57:18.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:57:18.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:57:18.965 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:57:19.191 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:57:19.237 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:57:19.237 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:57:19.269 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:57:19.270 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:57:19.271 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:57:19.312 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:57:19.339 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T08:57:19.341 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T08:57:19.379 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:57:20.414 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:59:20.472 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T08:59:20.511 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:59:21.546 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T08:59:21.566 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T08:59:21.566 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T08:59:21.566 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:59:21.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:59:21.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T08:59:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T08:59:22.058 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T08:59:22.058 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T08:59:22.094 INFO:teuthology.misc:Compressing logs... 2021-03-22T08:59:22.129 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T08:59:22.158 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T08:59:22.160 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T08:59:22.199 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T08:59:23.223 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:01:23.311 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:01:23.356 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:01:24.361 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:01:24.378 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:01:24.378 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:01:24.378 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:01:24.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:01:24.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:01:24.649 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:01:24.878 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:01:24.879 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:01:24.919 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:01:24.948 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:01:24.986 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T09:01:24.987 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T09:01:25.029 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:01:26.030 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:03:26.088 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:03:26.128 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:03:27.189 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:03:27.204 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:03:27.205 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:03:27.205 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:03:27.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:03:27.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:03:27.458 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:03:27.688 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:03:27.688 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:03:27.732 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:03:27.760 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:03:27.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T09:03:27.797 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T09:03:27.847 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:03:28.877 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:05:28.952 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:05:28.991 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:05:30.059 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:05:30.079 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:05:30.079 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:05:30.079 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:05:30.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:05:30.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:05:30.335 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:05:30.586 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:05:30.586 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:05:30.636 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:05:30.665 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:05:30.700 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T09:05:30.702 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T09:05:30.743 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:05:31.856 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:07:31.915 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:07:31.960 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:07:33.029 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:07:33.046 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:07:33.046 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:07:33.046 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:07:33.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:07:33.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:07:33.316 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:07:51.619 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T09:07:51.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:07:52.784 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:09:52.844 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:09:52.886 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:09:53.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:09:53.996 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:09:53.996 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:09:53.996 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:09:53.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:09:54.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:09:54.261 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:09:54.291 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:09:55.350 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:09:56.449 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:09:57.550 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:09:58.651 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:09:59.756 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:10:00.856 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:10:01.917 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:10:02.977 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:10:04.075 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:10:04.075 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T09:10:04.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:10:05.200 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:12:05.263 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:12:05.303 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:12:06.327 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:12:06.343 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:12:06.344 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:12:06.344 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:12:06.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:12:06.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:12:06.600 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:12:07.302 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T09:12:07.340 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:12:08.381 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:14:08.445 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:14:08.488 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:14:10.306 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:14:10.335 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:14:10.335 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:14:10.336 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:14:10.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:14:10.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:14:10.611 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:14:10.907 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:14:10.941 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:14:10.941 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:14:10.972 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:14:10.972 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:14:10.974 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:14:11.013 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:14:11.031 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T09:14:11.084 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:14:11.084 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:14:11.100 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:14:11.100 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:14:11.102 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:14:11.154 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:14:11.160 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T09:14:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:14:11.224 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:14:11.240 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:14:11.240 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:14:11.284 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:14:13.018 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:16:13.076 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:16:13.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:16:15.015 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:16:15.034 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:16:15.034 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:16:15.034 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:16:15.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:16:15.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:16:15.294 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:17:15.265 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T09:17:15.334 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:17:17.245 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:19:17.311 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:19:17.355 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:19:19.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:19:19.210 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:19:19.210 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:19:19.210 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:19:19.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:19:19.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:19:19.491 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:19:19.794 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:19:19.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:19:19.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:19:19.852 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:19:19.852 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:19:19.853 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:19:19.898 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:19:19.969 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:19:19.970 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:19:20.004 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:19:20.040 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:19:20.058 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T09:19:20.110 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:19:20.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:19:20.126 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:19:20.126 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:19:20.173 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:19:21.232 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:21:21.293 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:21:21.335 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:21:22.357 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:21:22.375 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:21:22.375 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:21:22.375 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:21:22.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:21:22.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:21:22.634 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:21:22.954 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:21:22.991 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:21:22.992 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:21:23.018 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:21:23.018 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:21:23.020 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:21:23.070 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:21:23.150 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:21:23.150 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:21:23.262 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:21:23.327 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T09:21:23.332 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:21:23.411 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:21:23.411 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:21:23.432 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:21:23.433 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:21:23.472 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:21:24.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:23:24.679 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:23:24.717 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:23:25.780 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:23:25.796 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:23:25.796 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:23:25.797 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:23:25.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:23:26.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:23:26.054 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:23:26.395 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:23:26.396 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:23:26.431 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:23:26.465 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:23:26.535 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:23:26.536 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:23:26.570 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:23:26.604 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:23:26.672 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:23:26.672 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:23:26.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:23:27.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:25:27.857 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:25:27.897 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:25:29.807 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:25:29.825 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:25:29.825 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:25:29.825 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:25:29.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:25:30.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:25:30.096 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:25:30.419 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:25:30.419 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:25:30.450 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:25:30.489 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:25:30.560 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:25:30.560 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:25:30.598 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:25:30.628 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:25:30.781 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.11.log: file size changed while zipping 2021-03-22T09:25:30.876 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2021-03-22T09:25:30.998 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.15.log: file size changed while zipping 2021-03-22T09:25:31.142 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2021-03-22T09:25:31.175 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:25:31.175 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:25:31.364 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:25:32.425 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:27:32.490 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:27:32.530 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:27:33.578 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:27:33.599 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:27:33.599 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:27:33.599 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:27:33.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:27:33.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:27:33.858 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:27:34.208 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:27:34.208 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:27:34.240 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:27:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:27:34.349 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:27:34.349 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:27:34.383 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:27:34.421 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:27:34.442 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T09:27:34.442 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2021-03-22T09:27:34.442 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.15.log.gz already exists; not overwritten 2021-03-22T09:27:34.443 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.7.log.gz already exists; not overwritten 2021-03-22T09:27:34.444 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T09:27:34.484 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:27:35.585 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:29:35.646 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:29:35.685 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:29:36.742 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:29:36.759 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:29:36.759 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:29:36.759 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:29:36.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:29:36.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:29:37.018 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:29:55.394 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T09:29:55.451 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:29:56.562 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:31:56.623 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:31:56.667 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:31:57.763 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:31:57.784 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:31:57.784 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:31:57.784 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:31:57.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:31:58.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:31:58.063 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:32:01.094 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T09:32:01.133 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:32:02.246 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:34:02.317 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:34:02.356 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:34:03.513 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:34:03.531 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:34:03.531 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:34:03.531 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:34:03.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:34:03.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:34:03.793 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:34:03.944 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:34:03.990 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:34:03.990 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:34:04.021 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:34:04.021 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:34:04.024 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:34:04.061 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:34:04.086 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T09:34:04.132 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:34:04.132 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:34:04.148 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:34:04.148 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:34:04.152 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:34:04.202 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:34:04.210 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T09:34:04.274 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:34:04.274 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:34:04.291 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:34:04.294 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:34:04.338 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:34:05.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:36:05.505 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:36:05.546 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:36:06.701 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:36:06.722 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:36:06.723 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:36:06.723 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:36:06.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:36:06.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:36:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:36:07.174 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:36:07.206 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:36:07.206 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:36:07.222 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:36:07.222 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:36:07.224 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:36:07.278 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:36:07.280 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T09:36:07.348 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:36:07.348 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:36:07.365 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:36:07.365 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:36:07.368 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:36:07.419 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:36:07.424 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T09:36:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:36:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:36:07.504 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:36:07.504 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:36:07.545 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:36:08.580 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:38:08.647 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:38:08.688 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:38:09.757 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:38:09.774 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:38:09.774 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:38:09.774 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:38:09.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:38:10.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:38:10.045 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:38:10.221 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:38:10.255 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:38:10.256 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:38:10.282 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:38:10.283 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:38:10.287 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:38:10.328 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:38:10.345 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T09:38:10.398 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:38:10.399 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:38:10.418 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:38:10.418 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:38:10.420 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:38:10.468 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:38:10.478 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T09:38:10.536 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:38:10.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:38:10.551 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:38:10.551 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:38:10.593 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:38:11.661 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:40:11.721 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:40:11.767 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:40:12.899 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:40:12.918 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:40:12.918 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:40:12.918 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:40:12.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:40:13.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:40:13.175 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:40:13.334 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:40:13.371 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:40:13.372 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:40:13.389 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:40:13.389 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:40:13.391 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:40:13.446 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:40:13.511 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:40:13.511 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:40:13.549 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:40:13.582 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T09:40:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:40:13.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:40:13.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:40:13.683 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:40:13.683 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:40:13.730 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:40:14.793 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:42:14.860 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:42:14.903 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:42:16.000 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:42:16.017 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:42:16.017 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:42:16.018 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:42:16.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:42:16.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:42:16.279 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:42:16.464 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:42:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:42:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:42:16.512 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:42:16.512 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:42:16.514 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:42:16.571 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:42:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:42:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:42:16.687 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:42:16.715 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:42:16.727 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T09:42:16.782 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:42:16.783 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:42:16.800 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:42:16.800 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:42:16.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:42:17.912 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:44:17.977 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:44:18.017 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:44:19.138 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:44:19.155 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:44:19.155 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:44:19.155 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:44:19.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:44:19.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:44:19.438 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:44:19.588 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:44:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:44:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:44:19.637 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:44:19.637 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:44:19.640 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:44:19.691 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:44:19.762 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:44:19.763 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:44:19.808 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:44:19.836 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:44:19.855 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T09:44:19.906 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:44:19.906 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:44:19.923 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:44:19.924 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:44:19.964 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:44:21.003 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:46:21.067 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:46:21.113 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:46:22.169 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:46:22.186 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:46:22.186 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:46:22.187 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:46:22.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:46:22.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:46:22.450 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:46:22.635 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:46:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:46:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:46:22.685 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:46:22.685 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:46:22.687 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:46:22.739 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:46:22.811 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:46:22.811 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:46:22.859 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:46:22.887 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:46:22.957 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T09:46:22.957 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:46:23.042 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:46:24.093 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:48:24.159 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:48:24.213 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:48:25.267 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:48:25.285 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:48:25.285 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:48:25.285 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:48:25.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:48:25.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:48:25.542 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:48:25.695 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:48:25.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:48:25.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:48:25.747 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:48:25.747 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:48:25.750 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:48:25.801 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:48:25.867 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:48:25.867 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:48:25.905 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:48:25.942 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T09:48:25.949 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:48:25.963 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T09:48:26.001 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:48:27.111 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:50:27.172 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:50:27.240 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:50:28.333 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:50:28.350 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:50:28.350 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:50:28.350 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:50:28.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:50:28.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:50:28.622 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:50:28.740 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:50:28.775 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:50:28.775 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:50:28.793 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:50:28.794 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:50:28.795 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:50:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:50:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:50:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:50:28.955 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:50:28.981 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T09:50:28.982 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/1ecf35f8-8af3-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T09:50:28.986 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:50:29.019 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T09:50:29.057 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:50:30.091 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:52:30.152 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:52:30.188 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:52:31.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:52:31.185 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:52:31.185 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:52:31.185 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:52:31.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:52:31.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:52:31.440 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:52:31.622 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T09:52:31.655 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:52:31.656 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:52:31.679 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T09:52:31.679 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:52:31.681 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:52:31.724 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T09:52:31.791 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T09:52:31.792 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T09:52:31.838 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:52:31.864 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T09:52:31.864 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/1ecf35f8-8af3-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T09:52:31.868 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T09:52:31.987 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T09:52:32.025 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:52:33.019 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:54:33.077 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:54:33.116 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:54:34.164 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:54:34.181 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:54:34.181 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:54:34.181 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:54:34.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:54:34.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:54:34.409 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T09:54:34.446 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:54:35.436 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:56:35.522 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:56:35.562 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:56:36.587 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:56:36.604 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:56:36.604 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:56:36.604 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:56:36.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:56:36.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:56:36.876 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:56:39.906 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T09:56:39.944 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:56:40.944 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:58:41.032 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T09:58:41.075 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:58:42.188 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T09:58:42.205 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T09:58:42.205 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T09:58:42.205 INFO:teuthology.misc:Compressing logs... 2021-03-22T09:58:42.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:58:42.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T09:58:42.461 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T09:58:42.489 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:58:43.586 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:58:44.682 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:58:45.780 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:58:46.879 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:58:47.981 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:58:49.041 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:58:50.138 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:58:51.198 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:58:52.299 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T09:58:52.300 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T09:58:52.399 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T09:58:53.469 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:00:53.540 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:00:53.579 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:00:54.626 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:00:54.643 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:00:54.643 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:00:54.643 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:00:54.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:00:54.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:00:54.898 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:01:10.348 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-22T10:01:10.349 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T10:01:10.387 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:01:11.432 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:03:11.493 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:03:11.533 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:03:12.594 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:03:12.612 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:03:12.613 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:03:12.613 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:03:12.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:03:12.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:03:12.884 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:03:13.165 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:03:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:03:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:03:13.225 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:03:13.226 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:03:13.228 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:03:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:03:13.282 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T10:03:13.331 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:03:13.331 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:03:13.347 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:03:13.347 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:03:13.350 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:03:13.398 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:03:13.404 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T10:03:13.466 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T10:03:13.466 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:03:13.483 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T10:03:13.483 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:03:13.521 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:03:14.578 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:05:14.642 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:05:14.685 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:05:15.731 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:05:15.748 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:05:15.748 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:05:15.749 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:05:15.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:05:15.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:05:16.004 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:05:48.202 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:05:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:05:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:05:48.259 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:05:48.259 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:05:48.261 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:05:48.306 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:05:48.319 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T10:05:48.375 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:05:48.375 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:05:48.391 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:05:48.391 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:05:48.394 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:05:48.443 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:05:48.451 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T10:05:48.511 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T10:05:48.511 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:05:48.526 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T10:05:48.527 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:05:48.567 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:05:49.640 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:07:49.704 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:07:49.743 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:07:50.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:07:50.904 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:07:50.905 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:07:50.905 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:07:50.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:07:51.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:07:51.166 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:07:51.450 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:07:51.483 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:07:51.483 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:07:51.514 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:07:51.514 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:07:51.517 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:07:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:07:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:07:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:07:51.661 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:07:51.692 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:07:51.713 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T10:07:51.760 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T10:07:51.760 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:07:51.775 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T10:07:51.775 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:07:51.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:07:53.030 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:09:53.097 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:09:53.136 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:09:54.856 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:09:54.872 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:09:54.873 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:09:54.873 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:09:54.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:09:55.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:09:55.147 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:09:55.423 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:09:55.455 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:09:55.455 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:09:55.484 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:09:55.484 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:09:55.487 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:09:55.529 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:09:55.599 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:09:55.600 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:09:55.639 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:09:55.668 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:09:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T10:09:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:09:55.801 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:09:56.842 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:11:56.912 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:11:56.950 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:11:58.073 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:11:58.090 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:11:58.090 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:11:58.090 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:11:58.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:11:58.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:11:58.359 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:11:58.708 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:11:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:11:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:11:58.770 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:11:58.771 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:11:58.773 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:11:58.817 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:11:58.891 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:11:58.891 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:11:58.936 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:11:58.974 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:11:59.045 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T10:11:59.045 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:11:59.109 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:12:00.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:14:00.258 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:14:00.295 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:14:01.342 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:14:01.359 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:14:01.359 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:14:01.359 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:14:01.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:14:01.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:14:01.614 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:14:01.907 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:14:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:14:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:14:01.972 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:14:01.972 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:14:01.974 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:14:02.012 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:14:02.084 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:14:02.084 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:14:02.129 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:14:02.158 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:14:02.229 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T10:14:02.229 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:14:02.298 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:14:03.325 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:16:03.381 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:16:03.421 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:16:04.548 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:16:04.565 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:16:04.565 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:16:04.565 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:16:04.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:16:04.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:16:04.836 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:16:05.122 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:16:05.155 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:16:05.155 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:16:05.181 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:16:05.181 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:16:05.184 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:16:05.225 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:16:05.295 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:16:05.295 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:16:05.339 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:16:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:16:05.396 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:16:05.396 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:16:05.437 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:16:06.478 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:18:06.544 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:18:06.586 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:18:07.673 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:18:07.691 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:18:07.691 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:18:07.691 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:18:07.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:18:07.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:18:07.952 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:18:08.239 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:18:08.271 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:18:08.272 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:18:08.294 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:18:08.294 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:18:08.296 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:18:08.342 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:18:08.414 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:18:08.414 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:18:08.458 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:18:08.485 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:18:08.523 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:18:08.524 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:18:08.524 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:18:08.564 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:18:10.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:20:10.192 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:20:10.370 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:20:11.457 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:20:11.477 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:20:11.477 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:20:11.477 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:20:11.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:20:11.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:20:11.738 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:20:12.041 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:20:12.074 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:20:12.075 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:20:12.107 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:20:12.108 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:20:12.109 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:20:12.144 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:20:12.215 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:20:12.215 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:20:12.250 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:20:12.283 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:20:12.333 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:20:12.334 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:20:12.358 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:20:12.360 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:20:12.397 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:20:13.449 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:22:13.506 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:22:13.548 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:22:14.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:22:14.660 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:22:14.660 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:22:14.660 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:22:14.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:22:14.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:22:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:22:15.216 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:22:15.248 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:22:15.248 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:22:15.277 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:22:15.278 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:22:15.281 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:22:15.319 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:22:15.393 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:22:15.393 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:22:15.434 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:22:15.461 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:22:15.490 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:22:15.491 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:22:15.491 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:22:15.493 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:22:15.531 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:22:16.556 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:24:16.615 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:24:16.652 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:24:18.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:24:18.609 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:24:18.610 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:24:18.610 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:24:18.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:24:18.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:24:18.869 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:24:19.148 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:24:19.179 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:24:19.179 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:24:19.200 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:24:19.200 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:24:19.204 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:24:19.252 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:24:19.323 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:24:19.324 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:24:19.368 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:24:19.395 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:24:19.411 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:24:19.411 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:24:19.411 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:24:19.411 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:24:19.411 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:24:19.411 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:24:19.413 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:24:19.453 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:24:20.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:26:20.601 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:26:20.645 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:26:22.663 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:26:22.680 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:26:22.680 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:26:22.680 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:26:22.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:26:22.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:26:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:26:23.246 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:26:23.278 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:26:23.278 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:26:23.310 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:26:23.310 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:26:23.312 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:26:23.348 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:26:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:26:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:26:23.453 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:26:23.489 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:26:23.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:26:23.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:26:23.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:26:23.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:26:23.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:26:23.510 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:26:23.512 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:26:23.552 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:26:24.593 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:28:24.657 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:28:24.697 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:28:26.487 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:28:26.503 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:28:26.503 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:28:26.503 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:28:26.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:28:26.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:28:26.773 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:28:27.060 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:28:27.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:28:27.091 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:28:27.118 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:28:27.118 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:28:27.122 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:28:27.162 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:28:27.232 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:28:27.232 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:28:27.267 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:28:27.300 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:28:27.319 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:28:27.320 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:28:27.320 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:28:27.320 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:28:27.320 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:28:27.320 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:28:27.320 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:28:27.361 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:28:28.368 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:30:28.427 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:30:28.465 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:30:30.399 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:30:30.417 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:30:30.417 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:30:30.417 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:30:30.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:30:30.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:30:30.673 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:30:30.973 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:30:31.003 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:30:31.004 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:30:31.034 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:30:31.034 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:30:31.035 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:30:31.075 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:30:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:30:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:30:31.186 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:30:31.214 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:30:31.214 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:30:31.214 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:30:31.214 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:30:31.214 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:30:31.215 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:30:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:30:31.219 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:30:31.257 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:30:32.312 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:32:32.374 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:32:32.423 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:32:33.489 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:32:33.506 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:32:33.506 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:32:33.506 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:32:33.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:32:33.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:32:33.769 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:32:34.069 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:32:34.101 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:32:34.101 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:32:34.131 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:32:34.131 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:32:34.132 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:32:34.171 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:32:34.241 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:32:34.242 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:32:34.282 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:32:34.310 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:32:34.341 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:32:34.341 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:32:34.341 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:32:34.341 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:32:34.342 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:32:34.342 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:32:34.343 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:32:34.381 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:32:35.467 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:34:35.530 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:34:35.628 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:34:36.653 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:34:36.669 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:34:36.669 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:34:36.669 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:34:36.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:34:36.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:34:36.936 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:34:37.280 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:34:37.314 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:34:37.314 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:34:37.336 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:34:37.336 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:34:37.341 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:34:37.383 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:34:37.451 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:34:37.451 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:34:37.506 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:34:37.537 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:34:37.560 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:34:37.560 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:34:37.560 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:34:37.560 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:34:37.560 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:34:37.560 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:34:37.561 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:34:37.600 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:34:38.608 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:36:38.667 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:36:38.705 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:36:39.772 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:36:39.788 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:36:39.788 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:36:39.789 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:36:39.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:36:40.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:36:40.050 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:36:40.368 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:36:40.400 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:36:40.400 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:36:40.423 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:36:40.424 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:36:40.425 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:36:40.472 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:36:40.546 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:36:40.546 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:36:40.573 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:36:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:36:40.629 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:36:40.630 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:36:40.630 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:36:40.630 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:36:40.630 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:36:40.630 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:36:40.631 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:36:40.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:36:41.678 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:38:41.736 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:38:41.774 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:38:42.827 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:38:42.846 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:38:42.846 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:38:42.846 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:38:42.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:38:43.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:38:43.104 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:38:44.459 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:38:44.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:38:44.496 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:38:44.529 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:38:44.529 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:38:44.536 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:38:44.567 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:38:44.638 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:38:44.638 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:38:44.693 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:38:44.728 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:38:44.729 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:38:44.729 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:38:44.730 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:38:44.730 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:38:44.730 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:38:44.730 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:38:44.732 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:38:44.768 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:38:45.794 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:40:45.853 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:40:45.895 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:40:46.949 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:40:46.966 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:40:46.966 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:40:46.966 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:40:46.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:40:47.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:40:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:40:47.664 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:40:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:40:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:40:47.729 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:40:47.730 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:40:47.731 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:40:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:40:47.838 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:40:47.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:40:47.877 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:40:47.909 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:40:47.929 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:40:47.929 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:40:47.929 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:40:47.929 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:40:47.930 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:40:47.930 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:40:47.931 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:40:47.969 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:40:49.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:42:49.084 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:42:49.126 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:42:50.289 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:42:50.306 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:42:50.306 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:42:50.306 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:42:50.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:42:50.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:42:50.564 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:42:51.163 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:42:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:42:51.196 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:42:51.243 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:42:51.243 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:42:51.245 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:42:51.274 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:42:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:42:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:42:51.380 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:42:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:42:51.431 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:42:51.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:42:51.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:42:51.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:42:51.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:42:51.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:42:51.432 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:42:51.472 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:42:52.537 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:44:52.603 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:44:52.643 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:44:53.670 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:44:53.688 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:44:53.688 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:44:53.688 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:44:53.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:44:53.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:44:53.945 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:44:54.228 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:44:54.262 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:44:54.262 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:44:54.286 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:44:54.286 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:44:54.288 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:44:54.331 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:44:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:44:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:44:54.445 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:44:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:44:54.497 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:44:54.497 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:44:54.497 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:44:54.497 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:44:54.497 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:44:54.498 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:44:54.499 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:44:54.537 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:44:55.582 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:46:55.640 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:46:55.681 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:46:56.795 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:46:56.813 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:46:56.813 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:46:56.814 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:46:56.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:46:57.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:46:57.094 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:46:57.362 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:46:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:46:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:46:57.422 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:46:57.423 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:46:57.426 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:46:57.467 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:46:57.536 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:46:57.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:46:57.576 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:46:57.607 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:46:57.632 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:46:57.632 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:46:57.632 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:46:57.632 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:46:57.632 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:46:57.633 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:46:57.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:46:57.672 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:46:58.786 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:48:58.853 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:48:58.892 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:48:59.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:48:59.887 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:48:59.887 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:48:59.887 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:48:59.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:49:00.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:49:00.148 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:49:00.437 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:49:00.469 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:49:00.469 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:49:00.495 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:49:00.495 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:49:00.498 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:49:00.542 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:49:00.614 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:49:00.614 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:49:00.647 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:49:00.682 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:49:00.708 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:49:00.708 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:49:00.708 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:49:00.709 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:49:00.709 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:49:00.709 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:49:00.709 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:49:00.749 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:49:01.728 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:51:01.786 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:51:01.823 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:51:02.936 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:51:02.957 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:51:02.957 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:51:02.958 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:51:02.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:51:03.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:51:03.227 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:51:03.590 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:51:03.624 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:51:03.624 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:51:03.647 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:51:03.647 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:51:03.649 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:51:03.698 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:51:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:51:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:51:03.811 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:51:03.845 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:51:03.851 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:51:03.852 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:51:03.852 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:51:03.852 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:51:03.852 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:51:03.852 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:51:03.853 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:51:03.894 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:51:04.948 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:53:05.008 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:53:05.048 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:53:06.072 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:53:06.090 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:53:06.091 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:53:06.091 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:53:06.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:53:06.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:53:06.373 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:53:06.652 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:53:06.684 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:53:06.684 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:53:06.714 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:53:06.714 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:53:06.715 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:53:06.756 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:53:06.827 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:53:06.827 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:53:06.862 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:53:06.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:53:06.920 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:53:06.920 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:53:06.920 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:53:06.920 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:53:06.920 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:53:06.921 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:53:06.922 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:53:06.959 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:53:07.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:55:08.035 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:55:08.072 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:55:09.145 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:55:09.162 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:55:09.162 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:55:09.162 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:55:09.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:55:09.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:55:09.419 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:55:09.712 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:55:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:55:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:55:09.771 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:55:09.772 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:55:09.775 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:55:09.824 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:55:09.894 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:55:09.894 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:55:09.936 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:55:09.964 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:55:09.988 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:55:09.988 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:55:09.988 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:55:09.988 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:55:09.988 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:55:09.988 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:55:09.991 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:55:10.029 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:55:11.041 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:57:11.105 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:57:11.144 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:57:12.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:57:12.248 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:57:12.248 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:57:12.248 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:57:12.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:57:12.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:57:12.518 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:57:12.817 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:57:12.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:57:12.849 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:57:12.879 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:57:12.880 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:57:12.882 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:57:12.918 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:57:12.987 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:57:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:57:13.034 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:57:13.062 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:57:13.091 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:57:13.092 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:57:13.092 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:57:13.092 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:57:13.092 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:57:13.092 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:57:13.093 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:57:13.129 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:57:14.182 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:59:14.242 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T10:59:14.286 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:59:15.365 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T10:59:15.382 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T10:59:15.383 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T10:59:15.383 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:59:15.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:59:15.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T10:59:15.666 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T10:59:15.943 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T10:59:15.975 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:59:15.975 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:59:15.999 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T10:59:15.999 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:59:16.000 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:59:16.046 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T10:59:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T10:59:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T10:59:16.149 INFO:teuthology.misc:Compressing logs... 2021-03-22T10:59:16.182 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T10:59:16.182 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T10:59:16.182 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T10:59:16.182 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T10:59:16.182 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T10:59:16.182 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T10:59:16.184 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T10:59:16.187 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T10:59:16.225 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T10:59:17.270 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:01:17.335 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:01:17.373 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:01:18.437 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:01:18.453 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:01:18.453 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:01:18.454 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:01:18.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:01:18.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:01:18.709 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:01:19.014 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:01:19.045 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:01:19.045 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:01:19.071 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:01:19.071 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:01:19.075 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:01:19.116 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:01:19.185 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:01:19.185 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:01:19.215 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:01:19.255 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:01:19.270 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:01:19.270 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:01:19.270 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:01:19.270 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:01:19.271 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:01:19.271 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:01:19.271 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:01:19.312 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:01:20.343 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:03:20.411 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:03:20.449 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:03:21.485 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:03:21.502 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:03:21.502 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:03:21.502 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:03:21.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:03:21.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:03:21.772 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:03:22.396 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:03:22.428 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:03:22.428 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:03:22.449 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:03:22.449 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:03:22.450 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:03:22.499 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:03:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:03:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:03:22.598 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:03:22.638 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:03:22.638 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:03:22.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:03:22.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:03:22.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:03:22.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:03:22.640 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:03:22.643 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:03:22.680 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:03:23.732 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:05:23.816 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:05:23.856 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:05:25.017 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:05:25.036 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:05:25.037 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:05:25.037 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:05:25.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:05:25.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:05:25.324 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:05:25.627 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:05:25.659 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:05:25.659 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:05:25.684 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:05:25.684 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:05:25.686 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:05:25.729 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:05:25.798 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:05:25.798 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:05:25.837 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:05:25.866 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:05:25.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:05:25.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:05:25.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:05:25.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:05:25.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:05:25.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:05:25.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:05:25.937 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:05:26.931 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:07:26.991 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:07:27.028 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:07:28.089 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:07:28.106 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:07:28.106 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:07:28.106 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:07:28.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:07:28.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:07:28.362 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:07:28.735 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:07:28.769 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:07:28.769 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:07:28.791 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:07:28.791 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:07:28.795 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:07:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:07:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:07:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:07:28.956 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:07:28.990 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:07:28.995 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:07:28.996 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:07:28.996 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:07:28.996 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:07:28.996 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:07:28.996 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:07:28.997 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:07:29.102 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:07:30.148 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:09:30.236 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:09:30.275 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:09:31.311 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:09:31.328 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:09:31.328 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:09:31.328 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:09:31.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:09:31.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:09:31.588 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:09:31.922 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:09:31.956 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:09:31.956 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:09:31.975 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:09:31.976 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:09:31.983 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:09:32.026 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:09:32.096 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:09:32.096 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:09:32.134 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:09:32.166 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:09:32.170 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:09:32.171 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:09:32.171 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:09:32.171 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:09:32.171 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:09:32.171 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:09:32.171 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:09:32.211 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:09:33.257 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:11:33.319 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:11:33.378 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:11:35.262 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:11:35.280 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:11:35.280 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:11:35.280 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:11:35.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:11:35.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:11:35.552 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:11:35.866 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:11:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:11:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:11:35.923 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:11:35.923 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:11:35.928 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:11:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:11:36.037 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:11:36.037 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:11:36.069 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:11:36.110 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:11:36.131 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:11:36.131 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:11:36.131 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:11:36.131 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:11:36.131 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:11:36.131 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:11:36.131 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:11:36.172 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:11:37.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:13:37.292 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:13:37.330 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:13:38.400 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:13:38.417 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:13:38.417 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:13:38.418 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:13:38.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:13:38.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:13:38.674 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:13:39.008 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:13:39.044 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:13:39.044 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:13:39.064 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:13:39.064 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:13:39.068 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:13:39.114 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:13:39.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:13:39.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:13:39.223 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:13:39.251 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:13:39.274 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:13:39.275 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:13:39.275 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:13:39.275 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:13:39.275 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:13:39.275 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:13:39.276 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:13:39.317 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:13:40.387 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:15:40.450 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:15:40.489 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:15:41.534 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:15:41.550 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:15:41.550 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:15:41.550 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:15:41.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:15:41.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:15:41.805 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:15:42.199 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:15:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:15:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:15:42.262 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:15:42.262 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:15:42.264 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:15:42.304 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:15:42.374 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:15:42.374 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:15:42.408 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:15:42.445 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:15:42.445 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:15:42.445 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:15:42.445 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:15:42.445 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:15:42.445 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:15:42.446 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:15:42.447 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:15:42.485 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:15:43.494 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:17:43.552 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:17:43.592 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:17:44.737 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:17:44.754 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:17:44.754 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:17:44.755 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:17:44.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:17:44.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:17:45.032 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:17:45.319 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:17:45.350 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:17:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:17:45.377 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:17:45.377 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:17:45.379 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:17:45.421 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:17:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:17:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:17:45.529 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:17:45.561 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:17:45.587 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:17:45.587 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:17:45.588 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:17:45.588 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:17:45.588 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:17:45.588 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:17:45.588 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:17:45.627 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:17:46.732 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:19:46.795 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:19:46.839 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:19:47.876 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:19:47.893 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:19:47.894 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:19:47.894 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:19:47.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:19:48.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:19:48.148 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:19:48.434 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:19:48.465 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:19:48.466 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:19:48.489 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:19:48.489 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:19:48.490 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:19:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:19:48.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:19:48.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:19:48.647 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:19:48.675 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:19:48.689 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:19:48.690 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:19:48.690 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:19:48.690 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:19:48.690 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:19:48.690 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:19:48.691 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:19:48.730 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:19:49.753 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:21:49.829 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:21:49.875 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:21:50.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:21:50.953 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:21:50.953 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:21:50.953 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:21:50.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:21:51.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:21:51.211 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:21:51.506 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:21:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:21:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:21:51.570 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:21:51.571 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:21:51.572 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:21:51.612 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:21:51.684 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:21:51.685 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:21:51.729 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:21:51.759 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:21:51.789 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:21:51.789 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:21:51.789 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:21:51.790 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:21:51.790 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:21:51.790 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:21:51.790 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:21:51.828 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:21:52.883 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:23:52.948 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:23:52.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:23:54.115 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:23:54.132 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:23:54.132 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:23:54.132 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:23:54.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:23:54.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:23:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:23:54.699 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:23:54.729 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:23:54.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:23:54.766 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:23:54.767 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:23:54.768 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:23:54.801 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:23:54.873 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:23:54.874 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:23:54.914 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:23:54.940 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:23:54.975 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:23:54.975 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:23:54.975 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:23:54.975 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:23:54.975 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:23:54.975 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:23:54.977 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:23:55.038 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:23:56.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:25:56.103 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:25:56.144 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:25:58.218 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:25:58.236 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:25:58.237 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:25:58.237 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:25:58.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:25:58.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:25:58.496 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:25:58.797 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:25:58.829 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:25:58.830 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:25:58.854 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:25:58.855 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:25:58.859 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:25:58.901 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:25:58.970 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:25:58.971 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:25:59.012 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:25:59.039 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:25:59.073 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:25:59.073 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:25:59.073 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:25:59.074 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:25:59.074 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:25:59.074 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:25:59.074 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:25:59.121 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:26:00.626 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:28:00.687 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:28:00.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:28:02.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:28:02.519 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:28:02.519 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:28:02.519 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:28:02.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:28:02.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:28:02.780 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:28:03.070 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:28:03.100 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:28:03.100 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:28:03.125 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:28:03.125 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:28:03.130 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:28:03.171 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:28:03.242 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:28:03.242 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:28:03.283 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:28:03.311 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:28:03.318 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:28:03.318 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:28:03.318 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:28:03.331 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:28:03.331 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:28:03.331 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:28:03.332 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:28:03.372 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:28:04.417 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:30:04.481 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:30:04.523 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:30:06.427 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:30:06.461 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:30:06.461 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:30:06.461 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:30:06.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:30:06.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:30:06.753 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:30:07.041 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:30:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:30:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:30:07.094 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:30:07.094 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:30:07.098 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:30:07.142 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:30:07.213 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:30:07.213 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:30:07.252 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:30:07.285 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:30:07.310 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:30:07.310 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:30:07.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:30:07.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:30:07.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:30:07.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:30:07.312 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:30:07.355 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:30:08.414 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:32:08.476 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:32:08.516 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:32:09.582 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:32:09.599 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:32:09.599 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:32:09.599 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:32:09.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:32:09.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:32:09.861 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:32:10.155 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:32:10.187 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:32:10.188 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:32:10.215 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:32:10.215 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:32:10.217 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:32:10.257 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:32:10.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:32:10.328 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:32:10.365 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:32:10.397 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:32:10.420 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:32:10.420 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:32:10.421 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:32:10.421 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/objectstore_tool.6.log.gz already exists; not overwritten 2021-03-22T11:32:10.421 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:32:10.421 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:32:10.421 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:32:10.421 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:32:10.467 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:32:11.512 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:34:11.575 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:34:11.613 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:34:13.455 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:34:13.485 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:34:13.485 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:34:13.485 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:34:13.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:34:13.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:34:13.753 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:34:14.064 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:34:14.097 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:34:14.097 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:34:14.124 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:34:14.124 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:34:14.126 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:34:14.168 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:34:14.238 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:34:14.239 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:34:14.278 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:34:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:34:14.335 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:34:14.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:34:14.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:34:14.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/objectstore_tool.6.log.gz already exists; not overwritten 2021-03-22T11:34:14.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:34:14.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:34:14.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:34:14.337 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:34:14.382 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:34:16.303 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:36:16.385 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:36:16.462 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:36:18.388 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:36:18.406 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:36:18.406 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:36:18.406 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:36:18.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:36:18.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:36:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:36:18.975 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:36:19.008 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:36:19.008 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:36:19.034 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:36:19.035 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:36:19.036 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:36:19.079 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:36:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:36:19.150 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:36:19.188 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:36:19.219 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:36:19.246 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:36:19.247 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:36:19.247 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:36:19.247 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/objectstore_tool.6.log.gz already exists; not overwritten 2021-03-22T11:36:19.247 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:36:19.247 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:36:19.247 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:36:19.248 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:36:19.289 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:36:20.330 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:38:20.391 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:38:20.431 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:38:21.512 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:38:21.529 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:38:21.529 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:38:21.529 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:38:21.529 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:38:21.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:38:21.791 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:38:22.096 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:38:22.129 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:38:22.130 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:38:22.160 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:38:22.160 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:38:22.162 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:38:22.200 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:38:22.272 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:38:22.272 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:38:22.313 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:38:22.342 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:38:22.371 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:38:22.371 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:38:22.371 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:38:22.371 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/objectstore_tool.6.log.gz already exists; not overwritten 2021-03-22T11:38:22.371 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:38:22.372 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:38:22.372 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:38:22.373 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:38:22.413 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:38:23.460 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:40:23.528 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:40:23.570 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:40:24.626 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:40:24.661 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:40:24.661 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:40:24.661 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:40:24.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:40:24.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:40:24.923 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:40:25.235 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:40:25.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:40:25.268 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:40:25.293 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:40:25.293 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:40:25.294 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:40:25.339 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:40:25.409 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:40:25.409 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:40:25.449 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:40:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:40:25.507 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:40:25.508 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:40:25.508 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:40:25.508 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/objectstore_tool.6.log.gz already exists; not overwritten 2021-03-22T11:40:25.508 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:40:25.508 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:40:25.508 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:40:25.509 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:40:25.552 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:40:26.645 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:42:26.710 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:42:26.752 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:42:27.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:42:27.910 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:42:27.910 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:42:27.911 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:42:27.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:42:28.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:42:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:42:28.459 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:42:28.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:42:28.491 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:42:28.519 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:42:28.520 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:42:28.524 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:42:28.562 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:42:28.634 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:42:28.634 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:42:28.679 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:42:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:42:28.721 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:42:28.722 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:42:28.722 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:42:28.722 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/objectstore_tool.6.log.gz already exists; not overwritten 2021-03-22T11:42:28.722 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:42:28.722 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:42:28.722 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:42:28.725 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:42:28.763 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:42:29.755 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:44:29.821 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:44:29.859 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:44:30.925 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:44:30.942 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:44:30.942 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:44:30.942 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:44:30.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:44:31.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:44:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:44:31.506 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:44:31.538 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:44:31.538 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:44:31.597 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:44:31.598 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:44:31.599 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:44:31.630 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:44:31.702 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:44:31.702 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:44:31.740 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:44:31.773 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:44:31.797 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:44:31.798 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:44:31.798 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:44:31.798 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/objectstore_tool.6.log.gz already exists; not overwritten 2021-03-22T11:44:31.798 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:44:31.798 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:44:31.798 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:44:31.799 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:44:31.839 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:44:32.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:46:33.004 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:46:33.055 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:46:34.119 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:46:34.137 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:46:34.137 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:46:34.137 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:46:34.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:46:34.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:46:34.402 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:46:34.707 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:46:34.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:46:34.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:46:34.765 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:46:34.765 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:46:34.768 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:46:34.808 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:46:34.877 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:46:34.877 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:46:34.921 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:46:34.948 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:46:34.980 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:46:34.981 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:46:34.981 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:46:34.981 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/objectstore_tool.6.log.gz already exists; not overwritten 2021-03-22T11:46:34.981 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:46:34.981 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:46:34.981 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:46:34.982 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:46:35.021 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:46:36.120 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:48:36.181 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:48:36.220 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:48:37.219 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:48:37.240 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:48:37.241 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:48:37.241 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:48:37.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:48:37.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:48:37.516 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:48:37.799 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:48:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:48:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:48:37.857 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:48:37.858 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:48:37.862 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:48:37.898 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:48:37.968 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:48:37.968 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:48:38.009 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:48:38.036 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:48:38.065 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:48:38.065 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:48:38.065 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:48:38.065 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/objectstore_tool.6.log.gz already exists; not overwritten 2021-03-22T11:48:38.065 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:48:38.065 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:48:38.065 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:48:38.068 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:48:38.106 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:48:39.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:50:39.202 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:50:39.241 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:50:40.301 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:50:40.319 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:50:40.320 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:50:40.320 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:50:40.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:50:40.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:50:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:50:40.875 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:50:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:50:40.908 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:50:40.935 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:50:40.935 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:50:40.938 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:50:40.978 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:50:41.051 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:50:41.051 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:50:41.088 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:50:41.120 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:50:41.148 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:50:41.148 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:50:41.148 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:50:41.148 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/objectstore_tool.6.log.gz already exists; not overwritten 2021-03-22T11:50:41.149 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:50:41.149 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:50:41.149 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:50:41.149 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:50:41.190 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:50:42.241 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:52:42.302 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:52:42.345 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:52:43.392 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:52:43.410 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:52:43.410 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:52:43.410 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:52:43.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:52:43.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:52:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:52:43.958 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T11:52:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:52:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:52:44.017 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T11:52:44.017 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:52:44.019 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:52:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T11:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T11:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T11:52:44.176 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:52:44.206 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T11:52:44.243 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-22T11:52:44.243 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-22T11:52:44.244 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-22T11:52:44.244 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/objectstore_tool.6.log.gz already exists; not overwritten 2021-03-22T11:52:44.244 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T11:52:44.244 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/dbfbdafc-8af6-11eb-90a1-001a4aab830c/ceph-osd.10.log.gz already exists; not overwritten 2021-03-22T11:52:44.244 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T11:52:44.245 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T11:52:44.290 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:52:45.301 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:54:45.366 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:54:45.405 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:54:46.476 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:54:46.497 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:54:46.497 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:54:46.498 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:54:46.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:54:46.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:54:46.769 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:55:05.282 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T11:55:05.322 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:55:06.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:57:06.426 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:57:06.465 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:57:07.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:57:07.539 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:57:07.540 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:57:07.540 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:57:07.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:57:07.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:57:07.799 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T11:57:24.037 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T11:57:24.073 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:57:25.141 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:59:25.201 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T11:59:25.243 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T11:59:26.347 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T11:59:26.366 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T11:59:26.366 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T11:59:26.366 INFO:teuthology.misc:Compressing logs... 2021-03-22T11:59:26.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:59:26.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T11:59:26.629 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:00:26.602 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T12:00:26.649 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:00:27.741 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:02:27.809 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:02:27.849 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:02:28.903 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:02:28.920 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:02:28.920 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:02:28.920 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:02:28.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:02:29.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:02:29.198 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:02:29.383 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:02:29.414 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:02:29.414 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:02:29.468 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:02:29.468 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:02:29.469 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:02:29.497 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:02:29.534 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:02:29.571 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:02:29.571 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:02:29.600 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:02:29.600 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:02:29.601 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:02:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:02:29.667 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:02:29.710 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:02:29.710 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:02:29.735 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:02:29.735 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:02:29.775 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:02:30.825 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:04:30.883 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:04:30.921 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:04:31.940 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:04:31.957 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:04:31.957 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:04:31.957 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:04:31.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:04:32.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:04:32.223 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:04:32.529 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:04:32.565 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:04:32.565 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:04:32.608 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:04:32.608 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:04:32.609 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:04:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:04:32.683 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:04:32.720 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:04:32.720 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:04:32.755 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:04:32.756 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:04:32.759 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:04:32.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:04:32.824 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:04:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:04:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:04:32.890 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:04:32.890 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:04:32.936 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:04:34.002 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:06:34.062 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:06:34.104 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:06:36.192 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:06:36.211 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:06:36.211 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:06:36.212 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:06:36.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:06:36.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:06:36.478 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:06:36.671 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:06:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:06:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:06:36.739 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:06:36.739 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:06:36.740 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:06:36.775 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:06:36.846 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:06:36.846 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:06:36.896 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:06:36.926 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:06:36.939 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:06:36.995 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:06:36.995 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:06:37.019 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:06:37.020 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:06:37.062 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:06:38.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:08:38.259 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:08:38.299 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:08:40.091 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:08:40.110 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:08:40.110 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:08:40.110 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:08:40.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:08:40.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:08:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:08:40.563 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:08:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:08:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:08:40.630 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:08:40.630 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:08:40.631 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:08:40.671 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:08:40.697 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:08:40.699 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:08:40.742 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:08:41.817 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:10:41.878 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:10:41.918 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:10:43.807 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:10:43.824 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:10:43.824 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:10:43.824 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:10:43.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:10:44.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:10:44.081 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:10:44.285 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:10:44.315 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:10:44.316 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:10:44.352 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:10:44.352 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:10:44.354 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:10:44.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:10:44.421 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:10:44.422 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:10:44.462 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:10:45.514 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:12:45.577 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:12:45.619 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:12:47.366 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:12:47.384 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:12:47.384 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:12:47.384 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:12:47.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:12:47.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:12:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:12:47.826 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:12:47.854 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:12:47.855 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:12:47.891 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:12:47.891 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:12:47.893 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:12:47.925 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:12:47.958 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:12:47.959 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:12:48.002 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:12:49.039 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:14:49.102 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:14:49.140 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:14:50.942 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:14:50.961 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:14:50.961 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:14:50.962 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:14:50.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:14:51.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:14:51.245 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:14:51.523 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:14:51.523 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:14:51.569 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:14:51.608 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:14:51.651 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:14:51.652 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:14:51.694 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:14:53.117 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:16:53.178 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:16:53.217 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:16:55.219 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:16:55.237 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:16:55.237 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:16:55.237 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:16:55.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:16:55.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:16:55.496 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:16:55.709 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:16:55.709 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:16:55.750 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:16:55.780 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:16:55.816 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:16:55.818 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:16:55.861 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:16:57.519 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:18:57.583 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:18:57.678 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:18:59.369 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:18:59.386 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:18:59.387 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:18:59.387 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:18:59.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:18:59.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:18:59.640 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:18:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:18:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:18:59.882 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:18:59.925 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:18:59.952 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:18:59.954 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:18:59.992 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:19:01.020 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:21:01.078 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:21:01.118 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:21:02.204 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:21:02.222 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:21:02.222 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:21:02.222 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:21:02.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:21:02.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:21:02.494 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:21:02.688 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:21:02.688 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:21:02.727 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:21:02.760 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:21:02.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:21:02.797 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:21:02.838 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:21:03.875 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:23:03.943 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:23:03.979 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:23:05.010 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:23:05.028 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:23:05.028 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:23:05.029 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:23:05.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:23:05.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:23:05.286 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:23:39.719 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T12:23:39.758 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:23:40.827 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:25:40.902 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:25:40.941 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:25:42.027 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:25:42.049 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:25:42.050 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:25:42.050 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:25:42.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:25:42.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:25:42.308 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:25:45.346 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T12:25:45.384 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:25:46.404 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:27:46.476 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:27:46.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:27:48.334 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:27:48.352 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:27:48.352 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:27:48.352 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:27:48.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:27:48.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:27:48.630 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:27:51.650 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T12:27:51.690 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:27:52.744 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:29:52.808 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:29:52.847 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:29:53.856 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:29:53.872 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:29:53.872 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:29:53.872 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:29:53.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:29:54.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:29:54.127 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:29:54.403 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:29:54.437 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:29:54.438 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:29:54.513 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:29:54.513 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:29:54.514 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:29:54.541 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:29:54.544 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:29:54.613 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:29:54.614 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:29:54.638 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:29:54.638 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:29:54.639 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:29:54.682 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:29:54.703 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:29:54.748 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:29:54.748 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:29:54.773 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:29:54.773 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:29:54.815 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:29:55.809 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:31:55.872 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:31:55.913 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:31:56.993 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:31:57.009 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:31:57.010 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:31:57.010 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:31:57.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:31:57.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:31:57.269 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:31:57.476 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:31:57.507 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:31:57.508 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:31:57.544 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:31:57.544 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:31:57.545 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:31:57.578 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:31:57.611 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:31:57.649 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:31:57.649 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:31:57.677 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:31:57.677 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:31:57.678 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:31:57.718 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:31:57.743 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:31:57.789 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:31:57.789 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:31:57.813 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:31:57.813 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:31:57.858 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:31:58.878 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:33:58.937 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:33:58.981 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:34:00.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:34:00.947 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:34:00.947 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:34:00.947 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:34:00.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:34:01.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:34:01.220 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:34:01.413 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:34:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:34:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:34:01.483 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:34:01.483 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:34:01.484 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:34:01.513 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:34:01.550 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:34:01.580 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:34:01.580 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:34:01.616 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:34:01.617 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:34:01.618 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:34:01.647 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:34:01.683 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:34:01.713 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:34:01.713 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:34:01.748 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:34:01.749 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:34:01.790 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:34:02.912 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:36:02.973 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:36:03.045 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:36:04.985 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:36:05.003 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:36:05.003 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:36:05.003 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:36:05.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:36:05.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:36:05.257 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:36:05.497 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:36:05.528 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:36:05.529 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:36:05.562 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:36:05.563 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:36:05.564 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:36:05.599 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:36:05.669 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T12:36:05.669 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:36:05.718 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:36:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:36:05.769 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T12:36:05.827 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:36:05.828 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:36:05.857 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T12:36:05.858 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:36:05.902 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:36:06.991 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:38:07.051 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:38:07.093 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:38:09.062 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:38:09.080 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:38:09.080 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:38:09.081 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:38:09.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:38:09.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:38:09.340 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:38:09.533 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:38:09.562 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:38:09.562 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:38:09.612 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:38:09.612 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:38:09.614 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:38:09.649 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:38:09.680 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:38:09.681 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:38:09.749 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:38:11.707 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:40:11.780 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:40:11.818 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:40:13.724 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:40:13.752 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:40:13.752 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:40:13.752 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:40:13.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:40:14.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:40:14.048 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:40:14.229 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:40:14.257 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:40:14.257 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:40:14.297 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:40:14.297 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:40:14.300 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:40:14.326 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:40:14.369 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:40:14.369 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:40:14.409 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:40:15.451 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:42:15.512 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:42:15.555 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:42:16.616 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:42:16.634 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:42:16.634 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:42:16.634 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:42:16.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:42:16.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:42:16.892 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:42:17.115 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:42:17.115 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:42:17.153 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:42:17.186 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:42:17.217 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:42:17.219 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:42:17.256 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:42:18.328 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:44:18.393 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:44:18.434 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:44:20.338 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:44:20.356 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:44:20.356 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:44:20.356 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:44:20.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:44:20.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:44:20.612 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:44:20.825 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:44:20.825 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:44:20.866 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:44:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:44:20.933 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:44:20.934 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:44:20.975 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:44:22.655 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:46:22.719 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:46:23.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:46:24.807 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:46:24.824 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:46:24.825 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:46:24.825 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:46:24.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:46:25.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:46:25.105 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:46:25.296 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:46:25.296 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:46:25.336 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:46:25.365 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:46:25.403 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:46:25.405 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:46:25.447 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:46:26.552 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:48:26.613 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:48:26.655 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:48:28.826 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:48:28.846 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:48:28.847 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:48:28.847 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:48:28.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:48:29.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:48:29.118 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:48:29.323 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:48:29.324 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:48:29.363 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:48:29.399 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:48:29.430 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:48:29.432 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:48:29.473 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:48:30.545 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:50:30.607 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:50:30.655 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:50:32.594 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:50:32.611 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:50:32.611 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:50:32.611 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:50:32.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:50:32.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:50:32.875 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:50:33.084 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:50:33.085 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:50:33.125 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:50:33.157 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:50:33.192 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:50:33.194 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:50:33.236 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:50:34.997 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:52:35.061 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:52:35.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:52:36.886 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:52:36.903 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:52:36.903 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:52:36.903 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:52:36.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:52:37.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:52:37.182 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:52:37.370 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:52:37.370 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:52:37.408 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:52:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:52:37.475 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:52:37.477 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:52:37.518 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:52:38.598 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:54:38.664 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:54:38.818 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:54:40.742 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:54:40.760 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:54:40.761 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:54:40.761 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:54:40.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:54:40.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:54:41.015 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:54:41.238 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:54:41.238 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:54:41.283 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:54:41.319 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:54:41.349 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:54:41.351 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:54:41.392 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:54:42.544 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:56:42.626 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:56:42.665 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:56:44.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:56:44.486 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:56:44.486 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:56:44.486 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:56:44.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:56:44.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:56:44.744 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:56:44.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:56:44.952 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:56:44.993 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:56:45.025 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:56:45.061 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:56:45.063 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:56:45.106 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:56:46.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:58:46.258 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T12:58:46.298 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:58:47.431 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T12:58:47.449 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T12:58:47.449 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T12:58:47.449 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:58:47.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:58:47.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T12:58:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T12:58:47.920 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T12:58:47.920 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T12:58:47.963 INFO:teuthology.misc:Compressing logs... 2021-03-22T12:58:47.994 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T12:58:48.031 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T12:58:48.032 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T12:58:48.074 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T12:58:49.122 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:00:49.187 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:00:49.238 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:00:51.448 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:00:51.467 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:00:51.468 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:00:51.468 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:00:51.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:00:51.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:00:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:00:51.971 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:00:51.972 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:00:52.013 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:00:52.046 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:00:52.080 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:00:52.083 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:00:52.154 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:00:53.962 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:02:54.024 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:02:54.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:02:56.247 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:02:56.264 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:02:56.265 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:02:56.265 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:02:56.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:02:56.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:02:56.524 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:02:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:02:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:02:56.776 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:02:56.805 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:02:56.844 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:02:56.846 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:02:56.911 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:02:58.887 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:04:58.949 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:04:59.004 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:05:01.004 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:05:01.022 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:05:01.022 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:05:01.023 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:05:01.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:05:01.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:05:01.307 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:05:01.506 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:05:01.506 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:05:01.547 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:05:01.577 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:05:01.615 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:05:01.617 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:05:01.678 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:05:03.459 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:07:03.523 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:07:03.567 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:07:05.593 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:07:05.611 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:07:05.611 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:07:05.611 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:07:05.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:07:05.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:07:05.870 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:07:06.084 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:07:06.085 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:07:06.123 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:07:06.156 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:07:06.188 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:07:06.190 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:07:06.231 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:07:08.159 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:09:08.223 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:09:08.265 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:09:10.177 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:09:10.257 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:09:10.257 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:09:10.257 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:09:10.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:09:10.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:09:10.520 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:09:10.729 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:09:10.730 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:09:10.805 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:09:10.837 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:09:10.852 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:09:10.853 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:09:10.896 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:09:12.659 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:11:12.756 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:11:12.797 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:11:14.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:11:14.769 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:11:14.770 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:11:14.770 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:11:14.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:11:15.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:11:15.044 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:11:15.262 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:11:15.262 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:11:15.360 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:11:15.393 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:11:15.429 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:11:15.430 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:11:15.472 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:11:17.051 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:13:17.116 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:13:17.156 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:13:19.196 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:13:19.214 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:13:19.214 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:13:19.214 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:13:19.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:13:19.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:13:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:13:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:13:19.702 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:13:19.767 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:13:19.796 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:13:19.824 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:13:19.826 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:13:19.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:13:21.763 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:15:21.824 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:15:21.867 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:15:23.983 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:15:24.000 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:15:24.001 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:15:24.001 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:15:24.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:15:24.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:15:24.273 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:15:24.480 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:15:24.480 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:15:24.571 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:15:24.612 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:15:24.613 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:15:24.616 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:15:24.659 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:15:25.805 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:17:25.880 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:17:25.920 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:17:27.716 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:17:27.733 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:17:27.733 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:17:27.733 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:17:27.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:17:27.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:17:28.009 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:17:28.199 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:17:28.200 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:17:28.290 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:17:28.322 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:17:28.327 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:17:28.329 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:17:28.369 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:17:29.476 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:19:29.544 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:19:29.586 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:19:30.744 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:19:30.761 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:19:30.761 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:19:30.761 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:19:30.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:19:30.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:19:31.023 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:19:31.240 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:19:31.241 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:19:31.322 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:19:31.352 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:19:31.359 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:19:31.361 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:19:31.398 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:19:32.439 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:21:32.500 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:21:32.543 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:21:33.691 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:21:33.708 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:21:33.708 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:21:33.708 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:21:33.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:21:33.939 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:21:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:21:34.179 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:21:34.179 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:21:34.263 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:21:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:21:34.302 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:21:34.304 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:21:34.344 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:21:35.451 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:23:35.515 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:23:35.554 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:23:36.635 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:23:36.652 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:23:36.652 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:23:36.652 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:23:36.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:23:36.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:23:36.923 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:23:37.115 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:23:37.116 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:23:37.217 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:23:37.253 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:23:37.256 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:23:37.257 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:23:37.298 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:23:38.376 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:25:38.440 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:25:38.483 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:25:39.518 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:25:39.535 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:25:39.535 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:25:39.535 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:25:39.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:25:39.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:25:39.798 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:25:40.025 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:25:40.026 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:25:40.128 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:25:40.158 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:25:40.160 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:25:40.164 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:25:40.199 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:25:41.243 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:27:41.303 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:27:41.370 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:27:43.190 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:27:43.207 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:27:43.208 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:27:43.208 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:27:43.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:27:43.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:27:43.465 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:27:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:27:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:27:43.815 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:27:43.848 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:27:43.852 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:27:43.853 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:27:43.980 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:27:45.787 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:29:45.851 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:29:45.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:29:47.705 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:29:47.722 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:29:47.722 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:29:47.722 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:29:47.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:29:47.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:29:47.996 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:29:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:29:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:29:48.307 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:29:48.335 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:29:48.337 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T13:29:48.340 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:29:48.380 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:29:49.444 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:31:49.508 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:31:49.552 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:31:51.536 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:31:51.557 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:31:51.557 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:31:51.557 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:31:51.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:31:51.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:31:51.814 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:31:59.147 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T13:32:00.248 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T13:32:01.356 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T13:32:02.457 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T13:32:03.560 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T13:32:04.620 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T13:32:05.720 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T13:32:06.779 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T13:32:07.880 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T13:32:08.938 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T13:32:08.938 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T13:32:08.979 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:32:10.220 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:34:10.285 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:34:10.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:34:12.235 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:34:12.253 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:34:12.253 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:34:12.253 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:34:12.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:34:12.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:34:12.512 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:34:15.554 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T13:34:15.597 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:34:17.535 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:36:17.599 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:36:17.641 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:36:19.582 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:36:19.601 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:36:19.601 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:36:19.601 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:36:19.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:36:19.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:36:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:36:20.021 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T13:36:20.071 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:36:20.072 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:36:20.099 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:36:20.099 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:36:20.101 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:36:20.143 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:36:20.163 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T13:36:20.213 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:36:20.214 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:36:20.232 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:36:20.232 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:36:20.234 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:36:20.282 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:36:20.291 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T13:36:20.350 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T13:36:20.351 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:36:20.368 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T13:36:20.368 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:36:20.410 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:36:22.135 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:38:22.198 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:38:22.240 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:38:24.045 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:38:24.065 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:38:24.065 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:38:24.065 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:38:24.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:38:24.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:38:24.335 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:38:24.476 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T13:38:24.510 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:38:24.510 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:38:24.527 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:38:24.527 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:38:24.530 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:38:24.579 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:38:24.649 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:38:24.649 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:38:24.684 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:38:24.717 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T13:38:24.723 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:38:24.793 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T13:38:24.793 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:38:24.815 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T13:38:24.815 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:38:24.854 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:38:25.906 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:40:25.974 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:40:26.011 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:40:27.044 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:40:27.061 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:40:27.062 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:40:27.062 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:40:27.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:40:27.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:40:27.316 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:40:27.467 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T13:40:27.498 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:40:27.499 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:40:27.515 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:40:27.516 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:40:27.518 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:40:27.567 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:40:27.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:40:27.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:40:27.674 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:40:27.709 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T13:40:27.710 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:40:27.780 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T13:40:27.780 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:40:27.803 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T13:40:27.803 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:40:27.849 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:40:28.972 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:42:29.047 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:42:29.093 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:42:30.874 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:42:30.892 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:42:30.892 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:42:30.892 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:42:30.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:42:31.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:42:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:42:31.288 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T13:42:31.318 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:42:31.318 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:42:31.334 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:42:31.334 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:42:31.336 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:42:31.385 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:42:31.454 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:42:31.454 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:42:31.494 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:42:31.521 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:42:31.534 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T13:42:31.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T13:42:31.587 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:42:31.605 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T13:42:31.605 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:42:31.647 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:42:32.707 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:44:32.769 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:44:32.814 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:44:34.602 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:44:34.619 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:44:34.619 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:44:34.619 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:44:34.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:44:34.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:44:34.879 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:44:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:44:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:44:35.087 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:44:35.136 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:44:35.205 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:44:35.206 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:44:35.241 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:44:35.283 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:44:35.350 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T13:44:35.351 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:44:35.412 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:44:36.474 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:46:36.538 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:46:36.603 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:46:38.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:46:38.617 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:46:38.617 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:46:38.617 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:46:38.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:46:38.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:46:38.876 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:46:39.054 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:46:39.054 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:46:39.073 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:46:39.123 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:46:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:46:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:46:39.234 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:46:39.275 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:46:39.352 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.11.log: file size changed while zipping 2021-03-22T13:46:39.413 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2021-03-22T13:46:39.458 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.8.log: file size changed while zipping 2021-03-22T13:46:39.522 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T13:46:39.522 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:46:39.672 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:46:40.674 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:48:40.734 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:48:40.772 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:48:42.599 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:48:42.617 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:48:42.617 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:48:42.617 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:48:42.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:48:42.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:48:42.888 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:48:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:48:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:48:43.096 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:48:43.167 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:48:43.235 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:48:43.235 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:48:43.279 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:48:43.307 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T13:48:43.307 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T13:48:43.311 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:48:43.314 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:48:43.368 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:48:44.414 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:50:44.477 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:50:44.518 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:50:45.645 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:50:45.662 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:50:45.662 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:50:45.663 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:50:45.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:50:45.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:50:45.929 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:50:46.114 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:50:46.114 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:50:46.134 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:50:46.185 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:50:46.257 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:50:46.257 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:50:46.295 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:50:46.336 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T13:50:46.337 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T13:50:46.340 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:50:46.343 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:50:46.384 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:50:47.601 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:52:47.664 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:52:47.701 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:52:48.739 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:52:48.754 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:52:48.755 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:52:48.755 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:52:48.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:52:48.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:52:49.022 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:52:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:52:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:52:49.224 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:52:49.265 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:52:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:52:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:52:49.373 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:52:49.415 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:52:49.421 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T13:52:49.421 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T13:52:49.423 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:52:49.461 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:52:50.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:54:50.696 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:54:50.739 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:54:51.785 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:54:51.801 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:54:51.802 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:54:51.802 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:54:51.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:54:52.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:54:52.080 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:54:52.248 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:54:52.249 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:54:52.280 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:54:52.318 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:54:52.387 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:54:52.387 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:54:52.432 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:54:52.469 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T13:54:52.469 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T13:54:52.475 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:54:52.478 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:54:52.514 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:54:53.527 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:56:53.590 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:56:53.630 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:56:54.643 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:56:54.660 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:56:54.660 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:56:54.661 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:56:54.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:56:54.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:56:54.920 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:56:55.144 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:56:55.145 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:56:55.164 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:56:55.213 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:56:55.284 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:56:55.284 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:56:55.325 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:56:55.346 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T13:56:55.346 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T13:56:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:56:55.353 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:56:55.391 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:56:56.406 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:58:56.470 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T13:58:56.506 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:58:57.504 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T13:58:57.522 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T13:58:57.522 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T13:58:57.522 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:58:57.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:58:57.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T13:58:57.783 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T13:58:57.971 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T13:58:57.971 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:58:58.008 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:58:58.040 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T13:58:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T13:58:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T13:58:58.151 INFO:teuthology.misc:Compressing logs... 2021-03-22T13:58:58.187 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T13:58:58.187 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T13:58:58.194 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T13:58:58.197 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T13:58:58.236 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T13:58:59.304 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:00:59.366 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:00:59.405 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:01:00.562 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:01:00.578 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:01:00.578 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:01:00.579 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:01:00.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:01:00.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:01:00.855 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:01:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:01:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:01:01.046 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:01:01.093 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:01:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:01:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:01:01.200 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:01:01.236 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:01:01.236 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:01:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:01:01.243 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:01:01.279 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:01:02.304 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:03:02.363 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:03:02.460 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:03:04.227 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:03:04.244 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:03:04.245 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:03:04.245 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:03:04.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:03:04.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:03:04.513 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:03:04.735 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:03:04.735 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:03:04.756 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:03:04.805 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:03:04.876 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:03:04.876 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:03:04.919 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:03:04.939 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:03:04.939 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:03:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:03:04.949 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:03:04.988 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:03:06.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:05:06.111 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:05:06.152 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:05:07.971 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:05:07.989 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:05:07.989 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:05:07.989 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:05:07.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:05:08.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:05:08.258 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:05:08.479 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:05:08.479 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:05:08.502 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:05:08.550 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:05:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:05:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:05:08.654 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:05:08.682 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:05:08.682 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:05:08.690 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:05:08.693 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:05:08.740 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:05:09.784 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:07:09.847 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:07:09.921 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:07:11.806 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:07:11.823 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:07:11.823 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:07:11.824 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:07:11.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:07:12.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:07:12.102 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:07:12.291 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:07:12.291 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:07:12.323 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:07:12.361 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:07:12.432 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:07:12.432 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:07:12.491 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:07:12.518 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:07:12.521 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:07:12.522 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:07:12.523 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:07:12.564 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:07:13.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:09:13.856 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:09:13.896 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:09:15.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:09:15.986 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:09:15.986 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:09:15.986 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:09:15.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:09:16.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:09:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:09:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:09:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:09:16.516 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:09:16.563 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:09:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:09:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:09:16.680 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:09:16.701 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:09:16.701 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:09:16.702 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:09:16.708 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:09:16.711 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:09:16.751 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:09:18.205 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:11:18.267 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:11:18.307 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:11:20.129 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:11:20.148 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:11:20.148 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:11:20.148 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:11:20.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:11:20.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:11:20.408 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:11:20.631 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:11:20.631 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:11:20.652 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:11:20.701 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:11:20.770 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:11:20.771 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:11:20.815 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:11:20.834 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:11:20.834 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:11:20.834 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:11:20.841 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:11:20.843 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:11:20.886 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:11:21.923 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:13:21.988 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:13:22.027 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:13:23.112 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:13:23.129 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:13:23.129 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:13:23.129 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:13:23.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:13:23.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:13:23.397 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:13:23.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:13:23.601 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:13:23.622 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:13:23.670 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:13:23.751 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:13:23.751 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:13:23.804 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:13:23.836 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:13:23.841 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:13:23.841 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:13:23.841 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:13:23.843 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:13:23.881 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:13:24.995 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:15:25.057 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:15:25.099 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:15:27.130 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:15:27.148 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:15:27.148 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:15:27.148 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:15:27.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:15:27.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:15:27.415 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:15:27.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:15:27.635 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:15:27.657 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:15:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:15:27.776 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:15:27.777 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:15:27.818 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:15:27.838 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:15:27.839 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:15:27.839 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:15:27.844 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:15:27.846 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:15:27.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:15:29.717 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:17:29.781 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:17:29.846 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:17:31.783 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:17:31.801 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:17:31.801 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:17:31.801 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:17:31.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:17:32.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:17:32.081 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:17:32.318 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:17:32.318 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:17:32.341 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:17:32.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:17:32.456 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:17:32.456 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:17:32.499 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:17:32.521 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:17:32.522 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:17:32.522 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:17:32.528 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:17:32.530 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:17:32.572 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:17:34.032 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:19:34.095 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:19:34.135 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:19:35.973 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:19:35.991 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:19:35.991 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:19:35.991 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:19:35.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:19:36.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:19:36.264 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:19:36.470 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:19:36.470 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:19:36.491 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:19:36.538 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:19:36.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:19:36.608 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:19:36.665 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:19:36.689 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:19:36.689 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:19:36.689 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:19:36.691 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:19:36.693 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:19:36.734 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:19:37.818 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:21:37.881 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:21:38.137 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:21:39.189 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:21:39.208 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:21:39.209 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:21:39.209 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:21:39.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:21:39.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:21:39.481 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:21:39.707 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:21:39.707 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:21:39.728 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:21:39.776 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:21:39.847 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:21:39.847 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:21:39.884 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:21:39.909 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:21:39.909 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:21:39.909 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:21:39.916 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:21:39.919 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:21:39.958 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:21:41.008 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:23:41.074 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:23:41.284 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:23:42.312 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:23:42.329 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:23:42.329 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:23:42.329 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:23:42.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:23:42.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:23:42.590 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:23:42.767 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:23:42.767 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:23:42.788 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:23:42.835 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:23:42.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:23:42.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:23:42.937 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:23:42.966 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:23:42.966 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:23:42.966 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:23:42.971 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:23:42.973 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:23:43.013 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:23:44.014 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:25:44.075 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:25:44.114 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:25:45.199 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:25:45.216 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:25:45.216 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:25:45.216 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:25:45.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:25:45.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:25:45.491 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:25:45.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:25:45.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:25:45.681 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:25:45.733 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:25:45.801 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:25:45.801 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:25:45.841 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:25:45.882 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:25:45.882 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:25:45.882 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:25:45.884 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:25:45.886 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:25:45.924 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:25:46.948 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:27:47.009 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:27:47.051 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:27:48.173 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:27:48.190 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:27:48.190 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:27:48.190 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:27:48.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:27:48.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:27:48.551 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:27:48.791 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:27:48.791 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:27:48.835 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:27:48.869 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:27:48.947 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:27:48.947 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:27:49.014 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:27:49.051 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:27:49.052 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:27:49.052 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:27:49.058 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:27:49.060 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:27:49.099 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:27:50.127 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:29:50.190 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:29:50.230 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:29:51.320 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:29:51.338 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:29:51.338 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:29:51.338 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:29:51.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:29:51.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:29:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:29:51.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:29:51.808 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:29:51.829 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:29:51.879 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:29:51.951 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:29:51.951 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:29:52.000 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:29:52.038 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:29:52.041 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:29:52.041 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:29:52.042 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:29:52.043 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:29:52.101 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:29:53.207 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:31:53.266 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:31:53.306 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:31:55.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:31:55.185 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:31:55.185 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:31:55.185 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:31:55.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:31:55.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:31:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:31:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:31:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:31:55.705 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:31:55.751 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:31:55.818 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:31:55.818 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:31:55.852 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:31:55.881 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:31:55.881 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:31:55.881 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:31:55.886 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:31:55.888 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:31:55.928 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:31:57.758 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:33:57.823 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:33:57.865 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:34:00.008 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:34:00.028 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:34:00.028 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:34:00.028 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:34:00.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:34:00.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:34:00.290 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:34:00.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:34:00.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:34:00.542 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:34:00.590 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:34:00.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:34:00.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:34:00.702 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:34:00.730 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:34:00.742 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:34:00.743 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:34:00.743 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:34:00.744 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:34:00.785 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:34:02.660 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:36:02.728 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:36:02.770 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:36:04.647 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:36:04.664 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:36:04.665 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:36:04.665 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:36:04.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:36:04.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:36:04.924 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:36:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:36:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:36:05.133 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:36:05.186 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:36:05.256 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:36:05.256 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:36:05.299 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:36:05.338 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:36:05.338 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:36:05.339 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:36:05.339 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:36:05.342 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:36:05.472 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:36:07.091 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:38:07.152 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:38:07.211 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:38:09.168 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:38:09.201 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:38:09.201 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:38:09.201 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:38:09.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:38:09.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:38:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:38:09.647 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:38:09.647 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:38:09.684 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:38:09.716 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:38:09.785 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:38:09.786 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:38:09.821 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:38:09.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:38:09.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:38:09.854 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:38:09.857 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:38:09.860 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:38:09.898 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:38:11.709 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:40:11.780 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:40:11.857 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:40:13.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:40:13.993 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:40:13.994 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:40:13.994 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:40:13.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:40:14.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:40:14.255 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:40:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:40:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:40:14.465 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:40:14.515 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:40:14.582 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:40:14.582 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:40:14.624 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:40:14.665 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:40:14.665 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:40:14.665 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:40:14.669 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:40:14.672 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:40:14.722 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:40:16.653 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:42:16.718 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:42:16.793 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:42:18.780 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:42:18.799 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:42:18.800 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:42:18.800 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:42:18.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:42:19.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:42:19.067 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:42:19.288 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:42:19.288 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:42:19.308 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:42:19.358 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:42:19.428 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:42:19.428 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:42:19.470 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:42:19.490 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:42:19.490 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:42:19.491 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:42:19.498 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:42:19.500 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:42:19.547 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:42:21.158 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:44:21.226 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:44:21.273 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:44:23.035 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:44:23.052 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:44:23.052 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:44:23.052 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:44:23.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:44:23.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:44:23.327 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:44:23.536 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:44:23.536 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:44:23.557 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:44:23.604 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:44:23.673 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:44:23.673 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:44:23.719 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:44:23.745 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:44:23.758 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:44:23.758 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:44:23.759 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:44:23.760 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:44:23.800 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:44:24.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:46:24.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:46:24.951 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:46:26.897 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:46:26.915 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:46:26.915 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:46:26.916 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:46:26.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:46:27.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:46:27.173 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:46:27.368 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:46:27.368 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:46:27.395 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:46:27.438 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:46:27.510 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:46:27.510 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:46:27.552 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:46:27.592 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:46:27.593 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:46:27.593 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:46:27.593 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:46:27.595 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:46:27.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:46:28.823 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:48:28.884 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:48:28.923 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:48:30.799 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:48:30.816 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:48:30.816 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:48:30.817 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:48:30.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:48:31.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:48:31.077 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:48:31.302 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:48:31.303 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:48:31.322 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:48:31.373 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:48:31.445 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:48:31.445 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:48:31.490 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:48:31.511 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:48:31.511 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:48:31.511 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:48:31.519 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:48:31.522 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:48:31.564 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:48:32.630 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:50:32.690 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:50:32.752 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:50:34.795 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:50:34.812 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:50:34.813 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:50:34.813 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:50:34.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:50:35.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:50:35.090 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:50:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:50:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:50:35.300 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:50:35.344 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:50:35.412 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:50:35.413 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:50:35.455 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:50:35.494 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:50:35.494 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:50:35.495 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:50:35.496 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:50:35.498 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:50:35.587 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:50:37.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:52:38.041 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:52:38.081 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:52:39.896 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:52:39.913 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:52:39.913 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:52:39.913 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:52:39.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:52:40.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:52:40.179 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:52:40.407 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:52:40.408 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:52:40.428 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:52:40.476 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:52:40.544 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:52:40.545 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:52:40.578 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:52:40.606 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:52:40.606 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:52:40.606 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:52:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:52:40.614 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:52:40.654 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:52:41.735 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:54:41.812 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:54:41.853 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:54:42.862 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:54:42.880 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:54:42.880 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:54:42.880 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:54:42.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:54:43.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:54:43.142 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:54:43.324 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:54:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:54:43.346 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:54:43.395 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:54:43.465 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:54:43.465 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:54:43.507 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:54:43.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:54:43.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:54:43.546 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:54:43.552 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:54:43.555 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:54:43.596 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:54:44.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:56:44.601 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:56:44.639 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:56:45.654 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:56:45.671 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:56:45.671 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:56:45.671 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:56:45.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:56:45.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:56:45.942 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:56:46.111 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:56:46.112 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:56:46.132 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:56:46.180 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:56:46.247 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:56:46.247 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:56:46.287 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:56:46.324 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:56:46.324 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:56:46.324 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:56:46.327 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:56:46.329 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:56:46.364 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:56:47.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:58:47.404 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T14:58:47.444 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:58:49.213 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T14:58:49.231 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T14:58:49.231 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T14:58:49.231 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:58:49.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:58:49.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T14:58:49.492 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T14:58:49.667 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T14:58:49.667 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:58:49.688 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:58:49.738 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T14:58:49.809 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T14:58:49.809 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T14:58:49.851 INFO:teuthology.misc:Compressing logs... 2021-03-22T14:58:49.887 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T14:58:49.887 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T14:58:49.887 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T14:58:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T14:58:49.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T14:58:49.937 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T14:58:50.978 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:00:51.041 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:00:51.082 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:00:53.072 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:00:53.092 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:00:53.092 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:00:53.092 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:00:53.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:00:53.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:00:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:00:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:00:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:00:53.606 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:00:53.654 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:00:53.725 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:00:53.725 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:00:53.762 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:00:53.787 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:00:53.787 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:00:53.787 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:00:53.792 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:00:53.795 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:00:53.838 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:00:55.759 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:02:55.821 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:02:55.867 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:02:58.292 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:02:58.311 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:02:58.312 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:02:58.312 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:02:58.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:02:58.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:02:58.598 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:02:58.774 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:02:58.774 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:02:58.802 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:02:58.845 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:02:58.915 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:02:58.915 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:02:58.955 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:02:58.983 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:02:58.984 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:02:58.984 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:02:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:02:58.993 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:02:59.033 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:03:00.114 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:05:00.179 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:05:00.222 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:05:02.336 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:05:02.357 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:05:02.357 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:05:02.357 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:05:02.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:05:02.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:05:02.625 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:05:02.810 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:05:02.810 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:05:02.842 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:05:02.884 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:05:02.955 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:05:02.955 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:05:02.996 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:05:03.046 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:05:03.047 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:05:03.047 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:05:03.053 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:05:03.056 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:05:03.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:05:04.539 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:07:04.606 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:07:04.646 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:07:06.623 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:07:06.643 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:07:06.643 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:07:06.643 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:07:06.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:07:06.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:07:06.905 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:07:07.081 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:07:07.081 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:07:07.102 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:07:07.149 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:07:07.217 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:07:07.217 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:07:07.256 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:07:07.289 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:07:07.291 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:07:07.291 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:07:07.291 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:07:07.292 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:07:07.330 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:07:08.331 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:09:08.398 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:09:08.439 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:09:09.482 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:09:09.501 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:09:09.501 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:09:09.501 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:09:09.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:09:09.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:09:09.776 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:09:09.987 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:09:09.987 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:09:10.006 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:09:10.057 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:09:10.134 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:09:10.135 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:09:10.179 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:09:10.200 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:09:10.201 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:09:10.201 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:09:10.206 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:09:10.208 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:09:10.248 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:09:11.273 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:11:11.328 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:11:11.371 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:11:12.389 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:11:12.482 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:11:12.483 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:11:12.483 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:11:12.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:11:12.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:11:12.735 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:11:12.959 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:11:12.960 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:11:12.980 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:11:13.031 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:11:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:11:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:11:13.147 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:11:13.167 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:11:13.167 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:11:13.167 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:11:13.174 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:11:13.177 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:11:13.214 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:11:14.256 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:13:14.317 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:13:14.356 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:13:15.424 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:13:15.443 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:13:15.443 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:13:15.443 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:13:15.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:13:15.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:13:15.704 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:13:15.897 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:13:15.897 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:13:15.918 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:13:15.968 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:13:16.038 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:13:16.038 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:13:16.084 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:13:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:13:16.122 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:13:16.122 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:13:16.122 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:13:16.123 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:13:16.162 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:13:17.224 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:15:17.284 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:15:17.326 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:15:18.408 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:15:18.426 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:15:18.426 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:15:18.426 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:15:18.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:15:18.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:15:18.700 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:15:18.868 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:15:18.868 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:15:18.898 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:15:18.941 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:15:19.007 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:15:19.007 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:15:19.046 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:15:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:15:19.090 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:15:19.091 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:15:19.091 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:15:19.092 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:15:19.130 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:15:20.123 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:17:20.182 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:17:20.218 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:17:21.240 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:17:21.260 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:17:21.260 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:17:21.260 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:17:21.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:17:21.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:17:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:17:21.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:17:21.739 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:17:21.759 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:17:21.812 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:17:21.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:17:21.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:17:21.925 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:17:21.946 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:17:21.946 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:17:21.946 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:17:21.953 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:17:21.956 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:17:21.992 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:17:23.036 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:19:23.097 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:19:23.139 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:19:25.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:19:25.250 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:19:25.251 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:19:25.251 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:19:25.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:19:25.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:19:25.518 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:19:25.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:19:25.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:19:25.763 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:19:25.813 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:19:25.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:19:25.883 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:19:25.926 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:19:25.946 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:19:25.946 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:19:25.946 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:19:25.952 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:19:25.955 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:19:25.998 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:19:27.682 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:21:27.760 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:21:27.802 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:21:29.575 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:21:29.593 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:21:29.593 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:21:29.593 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:21:29.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:21:29.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:21:29.863 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:21:30.032 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:21:30.033 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:21:30.062 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:21:30.102 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:21:30.170 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:21:30.170 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:21:30.210 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:21:30.251 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:21:30.251 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:21:30.251 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:21:30.251 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:21:30.253 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:21:30.290 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:21:31.308 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:23:31.395 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:23:31.434 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:23:33.233 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:23:33.255 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:23:33.255 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:23:33.255 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:23:33.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:23:33.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:23:33.518 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:23:33.702 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:23:33.703 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:23:33.732 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:23:33.771 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:23:33.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:23:33.843 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:23:33.892 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:23:33.927 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:23:33.928 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:23:33.928 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:23:33.933 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:23:33.935 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:23:33.975 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:23:35.800 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:25:35.888 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:25:35.929 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:25:37.966 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:25:37.982 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:25:37.982 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:25:37.983 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:25:37.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:25:38.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:25:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:25:38.473 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:25:38.474 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:25:38.493 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:25:38.544 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:25:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:25:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:25:38.656 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:25:38.677 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:25:38.677 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:25:38.677 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:25:38.684 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:25:38.687 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:25:38.727 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:25:40.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:27:40.088 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:27:40.126 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:27:42.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:27:42.101 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:27:42.101 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:27:42.101 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:27:42.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:27:42.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:27:42.374 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:27:42.550 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:27:42.551 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:27:42.572 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:27:42.621 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:27:42.686 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:27:42.686 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:27:42.727 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:27:42.766 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:27:42.766 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:27:42.767 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:27:42.768 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:27:42.771 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:27:42.811 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:27:44.243 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:29:44.306 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:29:44.345 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:29:46.211 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:29:46.231 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:29:46.231 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:29:46.231 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:29:46.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:29:46.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:29:46.483 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:29:46.674 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:29:46.674 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:29:46.695 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:29:46.742 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:29:46.810 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:29:46.810 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:29:46.855 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:29:46.892 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:29:46.892 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:29:46.892 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:29:46.899 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:29:46.901 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:29:46.941 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:29:48.258 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:31:48.322 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:31:48.364 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:31:50.291 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:31:50.308 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:31:50.308 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:31:50.308 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:31:50.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:31:50.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:31:50.569 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:31:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:31:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:31:50.792 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:31:50.832 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:31:50.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:31:50.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:31:50.950 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:31:50.987 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:31:50.987 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:31:50.987 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:31:50.989 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:31:50.992 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:31:51.035 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:31:52.405 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:33:52.466 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:33:52.507 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:33:54.407 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:33:54.424 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:33:54.424 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:33:54.425 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:33:54.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:33:54.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:33:54.703 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:33:54.880 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:33:54.880 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:33:54.908 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:33:54.951 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:33:55.021 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:33:55.021 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:33:55.072 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:33:55.101 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:33:55.101 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:33:55.101 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:33:55.107 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:33:55.110 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:33:55.150 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:33:57.098 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:35:57.166 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:35:57.209 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:35:59.186 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:35:59.210 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:35:59.210 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:35:59.210 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:35:59.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:35:59.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:35:59.476 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:35:59.662 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:35:59.662 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:35:59.683 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:35:59.732 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:35:59.804 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:35:59.804 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:35:59.845 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:35:59.878 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:35:59.878 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:35:59.879 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:35:59.885 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:35:59.888 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:35:59.959 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:36:01.911 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:38:01.976 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:38:02.018 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:38:04.007 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:38:04.024 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:38:04.024 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:38:04.024 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:38:04.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:38:04.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:38:04.285 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:38:04.479 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:38:04.479 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:38:04.500 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:38:04.551 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:38:04.618 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:38:04.619 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:38:04.658 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:38:04.696 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:38:04.702 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:38:04.702 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:38:04.702 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:38:04.703 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:38:04.747 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:38:06.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:40:06.758 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:40:06.805 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:40:09.032 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:40:09.069 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:40:09.069 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:40:09.070 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:40:09.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:40:09.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:40:09.348 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:40:09.522 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:40:09.522 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:40:09.542 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:40:09.592 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:40:09.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:40:09.661 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:40:09.699 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:40:09.735 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:40:09.735 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:40:09.736 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:40:09.741 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:40:09.744 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:40:09.788 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:40:11.561 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:42:11.638 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:42:11.705 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:42:13.631 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:42:13.653 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:42:13.653 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:42:13.653 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:42:13.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:42:13.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:42:13.927 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:42:14.134 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:42:14.134 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:42:14.168 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:42:14.204 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:42:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:42:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:42:14.327 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:42:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:42:14.359 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:42:14.359 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:42:14.359 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:42:14.360 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:42:14.406 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:42:16.174 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:44:16.238 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:44:16.321 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:44:18.448 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:44:18.467 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:44:18.467 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:44:18.467 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:44:18.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:44:18.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:44:18.734 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:44:18.992 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:44:18.993 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:44:19.023 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:44:19.075 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:44:19.150 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:44:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:44:19.194 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:44:19.250 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:44:19.260 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:44:19.260 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:44:19.260 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:44:19.261 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:44:19.325 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:44:21.158 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:46:21.225 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:46:21.376 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:46:23.733 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:46:23.763 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:46:23.763 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:46:23.763 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:46:23.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:46:24.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:46:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:46:24.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:46:24.246 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:46:24.278 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:46:24.317 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:46:24.386 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:46:24.386 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:46:24.436 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:46:24.465 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:46:24.483 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:46:24.484 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:46:24.484 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:46:24.485 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:46:24.535 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:46:27.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:48:27.417 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:48:27.456 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:48:29.297 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:48:29.315 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:48:29.316 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:48:29.316 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:48:29.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:48:29.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:48:29.575 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:48:29.765 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:48:29.765 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:48:29.785 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:48:29.835 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:48:29.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:48:29.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:48:29.939 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:48:29.976 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:48:29.976 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:48:29.976 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:48:29.981 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:48:29.984 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:48:30.026 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:48:31.183 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:50:31.249 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:50:31.292 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:50:33.317 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:50:33.334 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:50:33.334 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:50:33.334 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:50:33.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:50:33.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:50:33.593 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:50:33.783 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:50:33.783 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:50:33.805 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:50:33.853 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:50:33.925 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:50:33.925 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:50:33.966 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:50:34.003 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:50:34.004 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:50:34.004 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:50:34.010 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:50:34.013 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:50:34.057 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:50:36.038 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:52:36.102 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:52:36.140 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:52:38.096 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:52:38.113 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:52:38.113 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:52:38.114 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:52:38.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:52:38.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:52:38.389 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:52:38.567 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:52:38.568 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:52:38.589 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:52:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:52:38.706 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:52:38.706 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:52:38.748 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:52:38.784 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:52:38.784 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:52:38.784 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:52:38.788 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:52:38.791 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:52:38.839 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:52:39.995 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:54:40.057 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:54:40.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:54:41.887 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:54:41.904 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:54:41.904 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:54:41.905 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:54:41.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:54:42.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:54:42.171 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:54:42.358 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:54:42.359 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:54:42.381 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:54:42.432 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:54:42.501 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:54:42.501 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:54:42.537 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:54:42.565 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:54:42.566 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:54:42.566 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:54:42.575 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:54:42.578 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:54:42.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:54:43.677 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:56:43.739 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:56:43.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:56:44.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:56:44.897 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:56:44.897 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:56:44.897 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:56:44.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:56:45.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:56:45.152 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:56:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:56:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:56:45.364 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:56:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:56:45.485 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:56:45.485 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:56:45.526 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:56:45.563 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:56:45.563 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:56:45.563 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:56:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:56:45.572 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:56:45.630 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:56:46.670 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:58:46.733 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T15:58:46.775 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:58:47.834 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T15:58:47.852 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T15:58:47.852 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T15:58:47.853 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:58:47.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:58:48.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T15:58:48.132 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T15:58:48.305 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T15:58:48.305 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:58:48.329 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:58:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T15:58:48.443 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T15:58:48.444 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T15:58:48.485 INFO:teuthology.misc:Compressing logs... 2021-03-22T15:58:48.524 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T15:58:48.524 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T15:58:48.524 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T15:58:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T15:58:48.533 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T15:58:48.574 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T15:58:49.640 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:00:49.717 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:00:49.757 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:00:50.862 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:00:50.879 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:00:50.880 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:00:50.880 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:00:50.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:00:51.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:00:51.151 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:00:51.331 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:00:51.331 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:00:51.351 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:00:51.402 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:00:51.468 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:00:51.468 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:00:51.503 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:00:51.545 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:00:51.550 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:00:51.550 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:00:51.550 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:00:51.551 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:00:51.592 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:00:52.708 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:00:52.774 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/ Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 498, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 463, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 320, in report_job response.raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/sage-2021-03-18_16:32:54-rados:cephadm:dashboard-cephadm-iscsi-test-distro-basic-smithi/jobs/5977701/ 2021-03-22T16:02:52.778 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:02:52.821 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:02:53.887 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:02:53.905 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:02:53.905 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:02:53.905 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:02:53.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:02:54.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:02:54.168 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:02:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:02:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:02:54.365 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:02:54.415 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:02:54.485 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:02:54.486 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:02:54.524 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:02:54.563 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:02:54.563 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:02:54.563 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:02:54.569 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:02:54.572 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:02:54.610 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:02:55.633 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:04:55.733 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:04:55.783 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:04:56.904 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:04:56.928 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:04:56.929 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:04:56.929 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:04:56.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:04:57.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:04:57.206 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:04:57.385 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:04:57.386 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:04:57.417 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:04:57.458 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:04:57.524 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:04:57.524 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:04:57.563 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:04:57.609 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:04:57.610 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:04:57.610 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:04:57.614 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:04:57.617 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:04:57.656 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:04:58.723 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:06:58.788 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:06:58.829 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:06:59.972 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:06:59.989 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:06:59.989 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:06:59.989 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:06:59.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:07:00.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:07:00.261 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:07:00.445 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:07:00.446 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:07:00.467 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:07:00.518 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:07:00.586 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:07:00.586 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:07:00.632 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:07:00.667 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:07:00.667 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:07:00.667 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:07:00.671 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:07:00.673 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:07:00.717 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:07:01.801 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:09:01.865 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:09:01.907 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:09:03.022 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:09:03.040 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:09:03.040 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:09:03.040 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:09:03.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:09:03.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:09:03.308 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:09:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:09:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:09:03.505 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:09:03.554 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:09:03.625 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:09:03.625 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:09:03.659 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:09:03.701 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:09:03.711 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:09:03.712 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:09:03.712 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:09:03.713 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:09:03.753 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:09:04.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:11:04.880 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:11:04.921 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:11:06.019 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:11:06.039 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:11:06.039 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:11:06.039 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:11:06.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:11:06.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:11:06.330 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:11:06.506 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:11:06.506 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:11:06.536 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:11:06.576 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:11:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:11:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:11:06.683 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:11:06.715 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:11:06.726 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:11:06.727 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:11:06.727 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:11:06.730 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:11:06.773 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:11:07.916 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:13:07.979 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:13:08.041 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:13:09.130 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:13:09.148 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:13:09.148 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:13:09.148 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:13:09.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:13:09.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:13:09.422 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:13:09.605 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:13:09.605 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:13:09.625 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:13:09.674 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:13:09.743 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:13:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:13:09.785 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:13:09.820 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:13:09.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:13:09.821 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:13:09.825 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:13:09.828 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:13:09.872 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:13:10.964 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:15:11.026 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:15:11.065 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:15:12.105 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:15:12.123 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:15:12.123 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:15:12.123 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:15:12.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:15:12.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:15:12.386 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:15:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:15:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:15:12.590 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:15:12.627 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:15:12.694 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:15:12.695 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:15:12.736 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:15:12.775 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:15:12.775 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:15:12.775 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:15:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:15:12.778 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:15:12.823 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:15:13.886 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:17:13.949 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:17:13.986 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:17:15.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:17:15.064 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:17:15.064 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:17:15.065 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:17:15.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:17:15.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:17:15.340 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:17:15.511 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:17:15.511 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:17:15.530 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:17:15.582 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:17:15.652 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:17:15.653 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:17:15.695 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:17:15.736 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:17:15.736 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:17:15.736 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:17:15.739 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:17:15.741 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:17:15.778 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:17:16.803 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:19:16.866 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:19:16.936 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:19:18.055 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:19:18.073 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:19:18.073 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:19:18.073 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:19:18.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:19:18.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:19:18.340 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:19:18.523 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:19:18.523 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:19:18.544 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:19:18.592 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:19:18.663 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:19:18.663 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:19:18.707 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:19:18.745 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:19:18.745 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:19:18.745 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:19:18.750 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:19:18.752 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:19:18.870 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:19:19.947 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:21:20.022 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:21:20.061 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:21:21.120 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:21:21.137 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:21:21.137 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:21:21.137 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:21:21.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:21:21.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:21:21.402 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:21:21.580 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:21:21.581 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:21:21.602 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:21:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:21:21.719 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:21:21.719 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:21:21.759 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:21:21.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:21:21.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:21:21.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:21:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:21:21.806 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:21:21.843 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:21:22.874 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:23:22.932 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:23:22.974 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:23:24.023 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:23:24.040 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:23:24.040 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:23:24.041 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:23:24.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:23:24.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:23:24.312 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:23:24.484 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:23:24.484 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:23:24.505 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:23:24.553 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:23:24.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:23:24.621 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:23:24.659 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:23:24.694 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:23:24.694 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:23:24.694 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:23:24.699 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:23:24.702 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:23:24.740 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:23:25.785 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:25:25.845 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:25:25.886 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:25:26.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:25:26.987 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:25:26.987 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:25:26.987 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:25:26.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:25:27.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:25:27.266 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T16:25:27.461 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T16:25:27.462 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:25:27.482 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:25:27.531 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T16:25:27.600 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T16:25:27.600 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T16:25:27.641 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:25:27.670 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T16:25:27.683 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T16:25:27.683 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T16:25:27.683 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T16:25:27.684 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T16:25:27.725 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:25:28.777 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:27:28.841 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:27:28.880 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:27:29.947 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:27:29.964 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:27:29.964 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:27:29.964 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:27:29.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:27:48.290 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:27:48.331 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:27:49.322 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:29:49.380 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:29:49.425 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:29:50.576 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:29:50.596 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:29:50.596 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:29:50.596 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:29:50.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:29:53.670 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:29:53.709 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:29:54.811 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:31:54.877 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:31:54.918 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:31:56.079 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:31:56.102 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:31:56.102 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:31:56.102 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:31:56.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:31:59.174 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:31:59.212 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:32:00.379 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:34:00.446 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:34:00.487 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:34:01.561 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:34:01.580 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:34:01.580 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:34:01.580 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:34:01.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:34:04.642 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:34:04.683 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:34:05.728 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:36:05.792 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:36:05.834 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:36:06.915 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:36:06.933 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:36:06.934 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:36:06.934 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:36:06.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:36:09.987 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:36:10.033 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:36:11.147 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:38:11.212 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:38:11.253 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:38:12.319 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:38:12.336 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:38:12.337 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:38:12.337 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:38:12.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:38:15.394 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:38:15.434 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:38:16.458 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:40:16.526 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:40:16.566 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:40:17.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:40:17.610 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:40:17.610 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:40:17.610 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:40:17.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:40:20.674 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:40:20.708 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:40:21.687 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:42:21.751 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:42:21.790 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:42:22.838 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:42:22.856 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:42:22.856 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:42:22.856 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:42:22.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:42:25.926 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:42:25.967 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:42:27.004 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:44:27.067 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:44:27.107 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:44:28.149 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:44:28.167 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:44:28.168 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:44:28.168 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:44:28.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:44:31.234 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:44:31.272 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:44:32.277 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:46:32.338 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:46:32.377 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:46:33.381 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:46:33.398 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:46:33.398 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:46:33.398 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:46:33.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:46:36.454 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:46:36.497 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:46:37.578 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:48:37.644 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:48:37.680 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:48:38.682 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:48:38.699 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:48:38.699 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:48:38.699 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:48:38.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:48:41.762 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:48:41.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:48:42.725 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:50:42.784 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:50:42.822 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:50:43.806 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:50:43.824 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:50:43.824 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:50:43.824 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:50:43.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:50:46.882 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:50:46.941 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:50:47.891 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:52:47.953 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:52:47.992 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:52:48.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:52:48.992 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:52:48.992 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:52:48.992 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:52:48.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:52:52.066 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:52:52.101 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:52:53.130 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:54:53.192 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:54:53.230 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:54:54.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:54:54.207 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:54:54.208 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:54:54.208 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:54:54.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:54:57.282 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:54:57.318 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:54:58.318 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:56:58.380 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:56:58.418 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:56:59.371 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:56:59.388 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:56:59.388 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:56:59.388 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:56:59.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:57:02.466 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:57:02.500 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:57:03.424 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:59:03.487 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T16:59:03.588 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:59:04.576 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T16:59:04.592 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T16:59:04.592 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T16:59:04.592 INFO:teuthology.misc:Compressing logs... 2021-03-22T16:59:04.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T16:59:07.650 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T16:59:07.687 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T16:59:08.741 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:01:08.803 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:01:08.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:01:09.850 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:01:09.869 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:01:09.869 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:01:09.869 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:01:09.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:01:12.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T17:01:12.970 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:01:14.029 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:03:14.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:03:14.129 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:03:15.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:03:15.197 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:03:15.197 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:03:15.197 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:03:15.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:03:18.274 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T17:03:18.310 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:03:19.354 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:05:19.414 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:05:19.460 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:05:20.525 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:05:20.542 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:05:20.542 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:05:20.542 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:05:20.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:05:23.618 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T17:05:23.654 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:05:24.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:07:24.790 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:07:24.838 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:07:26.713 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:07:26.730 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:07:26.730 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:07:26.730 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:07:26.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:07:29.794 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T17:07:29.833 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:07:31.227 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:09:31.287 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:09:31.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:09:33.052 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:09:33.068 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:09:33.069 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:09:33.069 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:09:33.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:09:36.130 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T17:09:36.166 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:09:37.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:11:37.490 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:11:37.524 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:11:38.867 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:11:38.883 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:11:38.883 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:11:38.883 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:11:38.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:11:41.954 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T17:11:41.991 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:11:42.923 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:13:42.981 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:13:43.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:13:45.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:13:45.041 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:13:45.041 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:13:45.041 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:13:45.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:13:48.098 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T17:13:48.137 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:13:49.904 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:15:50.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:15:50.067 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:15:52.161 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:15:52.180 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:15:52.180 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:15:52.181 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:15:52.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:15:55.234 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T17:15:55.284 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:15:57.686 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:17:57.752 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:17:57.794 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:17:59.608 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:17:59.626 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:17:59.627 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:17:59.627 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:17:59.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:17:59.755 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:18:00.814 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:18:01.911 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:18:03.008 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:18:04.107 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:18:05.207 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:18:06.311 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:18:07.410 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:18:08.508 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:18:09.606 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:18:09.606 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T17:18:09.643 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:18:10.623 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:20:10.684 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:20:10.725 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:20:12.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:20:12.518 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:20:12.518 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:20:12.518 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:20:12.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:20:12.598 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:20:13.656 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:20:14.715 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:20:15.812 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:20:16.910 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:20:17.970 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:20:19.065 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:20:20.160 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:20:21.260 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:20:22.327 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T17:20:22.327 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T17:20:22.365 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:20:23.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:22:23.402 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:22:23.445 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:22:25.296 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:22:25.312 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:22:25.313 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:22:25.313 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:22:25.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:22:25.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:22:25.633 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T17:22:25.845 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T17:22:25.846 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T17:22:25.870 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:22:25.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T17:22:25.968 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T17:22:25.968 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T17:22:26.004 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:22:26.028 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T17:22:26.028 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T17:22:26.028 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T17:22:26.034 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T17:22:26.037 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:22:26.078 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:22:27.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:24:27.204 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:24:27.245 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:24:28.324 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:24:28.343 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:24:28.343 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:24:28.343 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:24:28.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:24:28.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:24:28.597 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T17:24:28.798 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T17:24:28.798 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T17:24:28.819 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:24:28.869 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T17:24:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T17:24:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T17:24:28.979 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:24:29.011 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T17:24:29.011 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T17:24:29.011 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T17:24:29.019 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T17:24:29.022 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:24:29.069 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:24:30.160 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:26:30.284 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:26:30.329 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:26:31.422 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:26:31.441 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:26:31.441 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:26:31.441 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:26:31.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:26:31.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:26:31.711 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T17:26:31.898 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T17:26:31.898 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T17:26:31.919 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:26:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T17:26:32.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T17:26:32.042 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T17:26:32.084 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:26:32.117 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T17:26:32.117 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T17:26:32.117 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T17:26:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T17:26:32.126 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:26:32.172 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:26:33.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:28:33.398 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:28:33.452 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:28:35.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:28:35.218 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:28:35.219 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:28:35.219 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:28:35.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:28:35.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:28:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T17:28:35.705 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T17:28:35.706 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T17:28:35.737 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:28:35.934 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:28:35.959 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T17:28:35.959 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2021-03-22T17:28:35.959 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T17:28:35.963 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T17:28:35.966 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:28:36.015 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:28:38.737 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:30:38.902 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:30:38.969 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:30:41.385 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:30:41.457 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:30:41.457 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:30:41.457 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:30:41.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:30:41.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:30:41.759 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T17:30:41.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T17:30:41.979 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T17:30:42.016 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:30:42.048 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:30:42.050 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:30:42.098 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:30:43.829 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:32:43.932 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:32:43.974 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:32:45.038 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:32:45.056 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:32:45.056 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:32:45.056 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:32:45.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:32:45.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:32:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T17:32:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T17:32:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T17:32:45.521 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:32:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:32:45.573 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:32:45.615 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:32:46.667 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:34:46.732 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:34:46.773 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:34:47.857 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:34:47.946 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:34:47.946 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:34:47.946 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:34:47.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:34:48.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:34:48.167 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T17:34:48.363 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T17:34:48.364 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T17:34:48.389 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:34:48.431 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:34:48.432 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:34:48.472 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:34:49.551 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:36:49.664 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:36:49.707 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:36:50.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:36:50.833 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:36:50.834 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:36:50.834 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:36:50.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:36:51.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:36:51.323 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:36:51.373 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:36:51.374 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:36:51.416 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:36:52.499 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:38:52.588 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:38:52.628 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:38:53.670 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:38:53.686 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:38:53.686 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:38:53.686 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:38:53.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:38:53.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:38:54.166 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:38:54.216 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:38:54.218 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:38:54.253 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:38:55.234 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:40:55.299 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:40:55.336 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:40:56.329 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:40:56.350 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:40:56.351 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:40:56.351 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:40:56.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:40:56.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:40:56.796 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:40:56.838 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:40:56.839 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:40:56.874 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:40:57.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:42:57.953 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:42:57.995 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:42:59.084 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:42:59.102 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:42:59.103 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:42:59.103 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:42:59.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:42:59.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:42:59.553 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:42:59.594 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:42:59.596 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:42:59.638 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:43:00.671 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:45:00.738 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:45:00.779 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:45:01.940 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:45:01.961 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:45:01.961 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:45:01.961 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:45:01.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:45:02.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:45:02.397 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:45:02.444 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:45:02.447 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:45:02.491 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:45:03.538 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:47:03.603 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:47:03.645 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:47:04.773 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:47:04.795 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:47:04.796 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:47:04.796 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:47:04.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:47:04.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:47:05.258 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:47:05.308 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:47:05.311 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:47:05.352 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:47:06.424 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:49:06.480 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:49:06.528 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:49:07.660 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:49:07.682 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:49:07.682 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:49:07.682 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:49:07.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:49:07.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:49:08.130 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:49:08.179 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:49:08.180 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:49:08.229 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:49:09.408 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:51:09.469 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:51:09.513 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:51:10.634 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:51:10.651 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:51:10.651 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:51:10.651 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:51:10.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:51:10.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:51:11.102 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:51:11.151 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:51:11.152 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:51:11.192 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:51:12.316 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:53:12.381 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:53:12.430 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:53:13.535 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:53:13.558 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:53:13.558 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:53:13.559 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:53:13.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:53:13.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:53:14.053 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:53:14.103 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:53:14.105 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:53:14.145 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:53:15.240 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:55:15.306 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:55:15.345 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:55:16.530 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:55:16.547 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:55:16.547 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:55:16.547 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:55:16.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:55:16.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:55:17.039 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:55:17.094 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:55:17.095 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:55:17.151 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:55:18.257 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:57:18.319 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:57:18.361 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:57:19.511 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:57:19.527 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:57:19.528 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:57:19.528 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:57:19.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:57:19.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:57:19.976 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:57:20.023 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:57:20.025 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:57:20.066 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:57:21.146 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:59:21.215 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T17:59:21.255 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:59:22.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T17:59:22.361 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T17:59:22.361 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T17:59:22.362 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:59:22.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:59:22.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T17:59:22.813 INFO:teuthology.misc:Compressing logs... 2021-03-22T17:59:22.862 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T17:59:22.864 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T17:59:22.904 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T17:59:23.952 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:01:24.017 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:01:24.066 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:01:25.154 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:01:25.173 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:01:25.173 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:01:25.173 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:01:25.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:01:25.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:01:25.621 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:01:25.669 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:01:25.671 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:01:25.712 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:01:26.776 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:03:26.838 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:03:26.879 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:03:27.940 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:03:27.958 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:03:27.958 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:03:27.958 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:03:27.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:03:28.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:03:28.440 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:03:28.491 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:03:28.493 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:03:28.535 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:03:29.569 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:05:29.634 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:05:29.673 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:05:30.781 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:05:30.802 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:05:30.802 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:05:30.802 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:05:30.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:05:31.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:05:31.250 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:05:31.300 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:05:31.301 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:05:31.345 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:05:32.424 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:07:32.484 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:07:32.527 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:07:33.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:07:33.617 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:07:33.617 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:07:33.617 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:07:33.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:07:33.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:07:34.075 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:07:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:07:34.116 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:07:34.159 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:07:35.287 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:09:35.357 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:09:35.461 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:09:36.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:09:36.520 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:09:36.520 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:09:36.520 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:09:36.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:09:36.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:09:36.962 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:09:37.007 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:09:37.011 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:09:37.073 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:09:38.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:11:38.205 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:11:38.246 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:11:39.288 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:11:39.305 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:11:39.305 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:11:39.305 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:11:39.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:11:39.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:11:39.755 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:11:39.798 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:11:39.800 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:11:39.839 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:11:40.917 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:13:40.979 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:13:41.020 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:13:42.130 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:13:42.147 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:13:42.148 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:13:42.148 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:13:42.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:13:42.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:13:42.595 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:13:42.641 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:13:42.643 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:13:42.685 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:13:43.803 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:15:43.863 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:15:43.916 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:15:45.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:15:45.085 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:15:45.086 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:15:45.086 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:15:45.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:15:45.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:15:45.533 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:15:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:15:45.576 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:15:45.617 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:15:46.789 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:17:46.856 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:17:46.895 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:17:48.027 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:17:48.045 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:17:48.045 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:17:48.045 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:17:48.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:17:48.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:17:48.531 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:17:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:17:48.585 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:17:48.635 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:17:49.767 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:19:49.838 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:19:49.879 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:19:50.987 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:19:51.008 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:19:51.008 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:19:51.008 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:19:51.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:19:51.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:19:51.472 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:19:51.513 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:19:51.518 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:19:51.557 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:19:52.691 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:21:52.755 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:21:52.797 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:21:53.842 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:21:53.858 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:21:53.859 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:21:53.859 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:21:53.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:21:54.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:21:54.298 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:21:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:21:54.341 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:21:54.381 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:21:55.486 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:23:55.548 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:23:55.588 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:23:56.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:23:56.660 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:23:56.660 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:23:56.660 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:23:56.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:23:56.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:23:57.101 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:23:57.155 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:23:57.156 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:23:57.197 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:23:58.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:25:58.327 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:25:58.368 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:25:59.418 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:25:59.435 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:25:59.435 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:25:59.436 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:25:59.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:25:59.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:25:59.875 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:25:59.921 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:25:59.923 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:25:59.963 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:26:00.991 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:28:01.069 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:28:01.112 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:28:02.173 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:28:02.190 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:28:02.190 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:28:02.191 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:28:02.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:28:02.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:28:02.643 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:28:02.685 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:28:02.686 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:28:02.735 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:28:03.834 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:30:03.898 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:30:03.939 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:30:05.033 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:30:05.051 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:30:05.051 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:30:05.051 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:30:05.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:30:05.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:30:05.511 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:30:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:30:05.560 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:30:05.616 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:30:06.744 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:32:06.806 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:32:06.845 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:32:07.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:32:08.009 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:32:08.010 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:32:08.010 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:32:08.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:32:08.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:32:08.450 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:32:08.496 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:32:08.498 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:32:08.538 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:32:09.646 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:34:09.713 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:34:09.754 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:34:10.836 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:34:10.854 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:34:10.854 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:34:10.855 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:34:10.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:34:11.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:34:11.299 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:34:11.346 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:34:11.348 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:34:11.387 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:34:12.422 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:36:12.484 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:36:12.523 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:36:13.607 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:36:13.625 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:36:13.626 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:36:13.626 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:36:13.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:36:13.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:36:14.090 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:36:14.141 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:36:14.142 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:36:14.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:36:15.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:38:15.404 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:38:15.447 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:38:16.550 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:38:16.566 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:38:16.567 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:38:16.567 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:38:16.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:38:16.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:38:17.014 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:38:17.051 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:38:17.053 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:38:17.090 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:38:18.102 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:40:18.182 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:40:18.224 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:40:19.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:40:19.285 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:40:19.285 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:40:19.286 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:40:19.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:40:19.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:40:19.720 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:40:19.771 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:40:19.773 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:40:19.813 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:40:20.851 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:42:20.914 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:42:20.953 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:42:22.002 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:42:22.020 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:42:22.020 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:42:22.021 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:42:22.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:42:22.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:42:22.456 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:42:22.505 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:42:22.507 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:42:22.616 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:42:23.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:44:23.698 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:44:23.736 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:44:24.797 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:44:24.815 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:44:24.815 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:44:24.815 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:44:24.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:44:25.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:44:25.260 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:44:25.298 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:44:25.300 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:44:25.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:44:26.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:46:26.493 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:46:26.540 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:46:27.580 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:46:27.597 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:46:27.597 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:46:27.597 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:46:27.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:46:27.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:46:28.043 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:46:28.090 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:46:28.091 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:46:28.130 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:46:29.203 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:48:29.265 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:48:29.303 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:48:30.399 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:48:30.420 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:48:30.421 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:48:30.421 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:48:30.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:48:30.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:48:30.879 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:48:30.925 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:48:30.926 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:48:30.973 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:48:32.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:50:32.111 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:50:32.155 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:50:33.229 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:50:33.246 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:50:33.246 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:50:33.247 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:50:33.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:50:33.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:50:33.691 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:50:33.738 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:50:33.739 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:50:33.781 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:50:34.863 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:52:34.927 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:52:34.973 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:52:36.056 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:52:36.073 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:52:36.074 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:52:36.074 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:52:36.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:52:36.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:52:36.523 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:52:36.568 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:52:36.569 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:52:36.609 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:52:37.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:54:37.703 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:54:37.756 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:54:38.834 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:54:38.850 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:54:38.851 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:54:38.851 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:54:38.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:54:39.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:54:39.299 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:54:39.333 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:54:39.335 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:54:39.377 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:54:40.422 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:56:40.483 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:56:40.528 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:56:41.673 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:56:41.691 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:56:41.692 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:56:41.692 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:56:41.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:56:41.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:56:42.130 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:56:42.179 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:56:42.180 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:56:42.222 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:56:43.331 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:58:43.393 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T18:58:43.435 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:58:44.477 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T18:58:44.495 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T18:58:44.495 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T18:58:44.495 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:58:44.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:58:44.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T18:58:44.939 INFO:teuthology.misc:Compressing logs... 2021-03-22T18:58:44.983 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T18:58:44.984 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T18:58:45.024 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T18:58:46.069 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:00:46.132 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:00:46.172 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:00:47.213 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:00:47.234 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:00:47.235 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:00:47.235 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:00:47.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:00:47.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:00:47.683 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:00:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:00:47.725 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:00:47.770 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:00:48.847 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:02:48.907 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:02:48.949 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:02:50.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:02:50.862 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:02:50.863 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:02:50.863 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:02:50.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:02:51.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:02:51.315 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:02:51.365 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:02:51.367 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:02:51.407 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:02:52.498 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:04:52.569 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:04:52.611 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:04:53.664 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:04:53.682 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:04:53.682 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:04:53.682 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:04:53.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:04:53.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:04:54.133 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:04:54.176 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:04:54.177 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:04:54.220 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:04:55.242 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:06:55.308 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:06:55.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:06:56.420 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:06:56.438 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:06:56.438 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:06:56.438 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:06:56.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:06:56.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:06:56.885 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:06:56.930 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:06:56.932 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:06:56.976 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:06:58.016 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:08:58.093 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:08:58.136 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:08:59.189 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:08:59.206 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:08:59.206 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:08:59.207 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:08:59.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:08:59.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:08:59.654 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:08:59.701 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:08:59.702 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:08:59.741 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:09:00.836 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:11:00.901 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:11:00.958 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:11:03.474 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:11:03.512 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:11:03.513 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:11:03.513 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:11:03.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:11:03.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:11:03.974 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:11:04.024 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:11:04.026 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:11:04.074 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:11:05.215 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:13:05.289 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:13:05.333 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:13:07.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:13:07.643 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:13:07.643 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:13:07.644 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:13:07.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:13:07.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:13:08.108 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:13:08.149 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:13:08.154 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:13:08.219 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:13:10.857 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:15:10.924 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:15:10.981 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:15:12.092 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:15:12.108 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:15:12.109 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:15:12.109 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:15:12.109 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:15:12.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:15:12.552 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:15:12.602 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:15:12.604 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:15:12.653 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:15:13.761 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:17:13.826 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:17:13.950 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:17:15.039 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:17:15.056 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:17:15.056 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:17:15.056 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:17:15.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:17:15.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:17:15.500 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:17:15.540 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:17:15.542 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:17:15.582 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:17:16.671 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:19:16.732 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:19:16.774 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:19:17.840 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:19:17.858 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:19:17.858 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:19:17.858 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:19:17.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:19:18.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:19:18.302 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:19:18.339 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:19:18.341 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:19:18.384 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:19:19.520 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:21:19.582 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:21:19.622 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:21:20.700 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:21:20.718 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:21:20.718 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:21:20.718 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:21:20.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:21:20.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:21:21.159 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:21:21.198 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:21:21.200 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:21:21.240 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:21:22.257 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:23:22.321 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:23:22.368 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:23:23.478 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:23:23.495 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:23:23.495 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:23:23.496 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:23:23.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:23:23.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:23:23.938 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:23:23.986 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:23:23.989 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:23:24.029 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:23:25.070 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:25:25.132 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:25:25.172 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:25:26.294 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:25:26.312 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:25:26.312 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:25:26.312 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:25:26.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:25:26.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:25:26.752 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:25:26.797 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:25:26.799 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:25:26.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:25:27.897 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:27:27.966 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:27:28.006 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:27:29.053 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:27:29.070 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:27:29.070 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:27:29.071 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:27:29.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:27:29.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:27:29.498 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:27:29.548 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:27:29.549 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:27:29.588 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:27:30.644 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:29:30.714 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:29:30.759 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:29:31.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:29:31.919 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:29:31.919 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:29:31.920 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:29:31.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:29:32.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:29:32.364 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:29:32.405 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:29:32.406 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:29:32.445 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:29:33.470 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:31:33.537 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:31:33.582 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:31:34.717 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:31:34.735 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:31:34.735 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:31:34.735 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:31:34.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:31:34.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:31:35.176 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:31:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:31:35.224 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:31:35.267 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:31:36.352 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:33:36.418 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:33:36.462 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:33:37.518 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:33:37.537 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:33:37.537 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:33:37.537 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:33:37.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:33:37.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:33:37.985 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:33:38.030 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:33:38.031 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:33:38.074 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:33:39.127 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:35:39.194 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:35:39.238 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:35:40.336 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:35:40.353 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:35:40.353 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:35:40.353 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:35:40.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:35:40.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:35:40.801 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:35:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:35:40.847 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:35:40.892 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:35:42.017 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:37:42.081 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:37:42.122 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:37:43.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:37:43.300 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:37:43.300 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:37:43.300 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:37:43.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:37:43.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:37:43.744 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:37:43.785 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:37:43.787 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:37:43.829 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:37:44.860 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:39:44.922 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:39:44.966 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:39:45.981 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:39:45.998 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:39:45.998 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:39:45.999 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:39:45.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:39:46.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:39:46.472 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:39:46.521 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:39:46.522 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:39:46.562 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:39:47.676 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:41:47.740 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:41:47.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:41:48.836 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:41:48.853 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:41:48.853 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:41:48.854 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:41:48.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:41:49.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:41:49.298 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:41:49.342 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:41:49.344 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:41:49.382 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:41:50.415 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:43:50.481 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:43:50.524 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:43:51.705 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:43:51.722 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:43:51.722 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:43:51.722 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:43:51.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:43:51.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:43:52.173 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:43:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:43:52.217 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:43:52.260 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:43:53.392 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:45:53.461 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:45:53.502 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:45:54.491 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:45:54.507 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:45:54.507 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:45:54.507 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:45:54.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:45:54.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:45:54.938 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:45:54.985 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:45:54.986 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:45:55.024 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:45:55.987 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:47:56.054 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:47:56.095 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:47:57.123 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:47:57.140 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:47:57.141 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:47:57.141 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:47:57.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:47:57.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:47:57.568 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:47:57.617 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:47:57.618 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:47:57.656 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:47:58.638 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:49:58.699 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:49:58.739 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:49:59.832 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:49:59.850 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:49:59.850 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:49:59.851 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:49:59.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:50:00.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:50:00.306 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:50:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:50:00.360 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:50:00.467 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:50:01.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:52:01.555 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:52:01.594 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:52:02.672 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:52:02.689 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:52:02.690 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:52:02.690 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:52:02.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:52:02.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:52:03.173 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:52:03.223 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:52:03.224 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:52:03.263 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:52:04.319 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:54:04.406 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:54:04.448 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:54:05.599 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:54:05.620 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:54:05.621 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:54:05.621 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:54:05.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:54:05.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:54:06.069 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:54:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:54:06.122 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:54:06.164 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:54:07.266 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:56:07.329 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:56:07.370 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:56:08.430 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:56:08.448 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:56:08.448 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:56:08.449 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:56:08.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:56:08.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:56:08.931 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:56:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:56:08.985 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:56:09.038 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:56:10.070 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:58:10.132 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T19:58:10.172 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:58:11.252 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T19:58:11.269 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T19:58:11.270 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T19:58:11.270 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:58:11.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:58:11.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:58:11.710 INFO:teuthology.misc:Compressing logs... 2021-03-22T19:58:11.756 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T19:58:11.757 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T19:58:11.798 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T19:58:12.846 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:00:12.908 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:00:12.948 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:00:14.099 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:00:14.120 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:00:14.120 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:00:14.120 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:00:14.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:00:14.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:00:14.551 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:00:14.600 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:00:14.602 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:00:14.641 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:00:15.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:02:15.783 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:02:15.822 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:02:16.931 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:02:16.949 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:02:16.949 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:02:16.949 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:02:16.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:02:17.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:02:17.398 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:02:17.450 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:02:17.451 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:02:17.491 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:02:18.525 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:04:18.587 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:04:18.625 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:04:19.661 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:04:19.680 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:04:19.680 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:04:19.680 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:04:19.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:04:19.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:04:20.181 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:04:20.234 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:04:20.236 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:04:20.275 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:04:21.320 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:06:21.380 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:06:21.422 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:06:22.523 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:06:22.546 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:06:22.547 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:06:22.547 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:06:22.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:06:22.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:06:23.032 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:06:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:06:23.086 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:06:23.128 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:06:24.242 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:08:24.307 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:08:24.348 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:08:25.371 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:08:25.388 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:08:25.388 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:08:25.388 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:08:25.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:08:25.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:08:25.868 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:08:25.919 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:08:25.920 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:08:25.959 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:08:26.963 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:10:27.024 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:10:27.066 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:10:28.199 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:10:28.216 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:10:28.216 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:10:28.216 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:10:28.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:10:28.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:10:28.667 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:10:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:10:28.713 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:10:28.752 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:10:29.801 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:12:29.864 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:12:29.915 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:12:30.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:12:30.955 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:12:30.955 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:12:30.955 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:12:30.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:12:31.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:12:31.391 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:12:31.440 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:12:31.442 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:12:31.480 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:12:32.499 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:14:32.562 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:14:32.602 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:14:33.678 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:14:33.695 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:14:33.696 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:14:33.696 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:14:33.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:14:33.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:14:34.138 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:14:34.182 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:14:34.184 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:14:34.224 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:14:35.259 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:16:35.320 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:16:35.360 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:16:36.435 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:16:36.452 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:16:36.452 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:16:36.452 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:16:36.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:16:36.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:16:36.884 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:16:36.936 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:16:36.938 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:16:36.981 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:16:38.009 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:18:38.075 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:18:38.119 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:18:39.174 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:18:39.195 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:18:39.195 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:18:39.195 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:18:39.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:18:39.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:18:39.636 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:18:39.684 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:18:39.686 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:18:39.724 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:18:40.770 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:20:40.830 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:20:40.871 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:20:41.942 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:20:41.960 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:20:41.960 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:20:41.960 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:20:41.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:20:42.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:20:42.407 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:20:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:20:42.446 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:20:42.487 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:20:43.521 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:22:43.586 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:22:43.630 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:22:44.675 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:22:44.691 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:22:44.691 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:22:44.691 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:22:44.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:22:44.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:22:45.123 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:22:45.172 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:22:45.173 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:22:45.215 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:22:46.234 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:24:46.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:24:46.338 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:24:47.432 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:24:47.452 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:24:47.453 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:24:47.453 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:24:47.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:24:47.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:24:47.893 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:24:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:24:47.940 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:24:48.039 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:24:49.099 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:26:49.160 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:26:49.199 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:26:50.247 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:26:50.264 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:26:50.265 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:26:50.265 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:26:50.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:26:50.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:26:50.704 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:26:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:26:50.754 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:26:50.794 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:26:51.862 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:28:51.922 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:28:51.963 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:28:52.993 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:28:53.011 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:28:53.011 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:28:53.012 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:28:53.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:28:53.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:28:53.454 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:28:53.495 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:28:53.496 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:28:53.535 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:28:54.572 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:30:54.637 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:30:54.686 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:30:55.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:30:55.863 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:30:55.863 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:30:55.863 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:30:55.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:30:56.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:30:56.317 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:30:56.363 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:30:56.365 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:30:56.405 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:30:57.521 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:32:57.585 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:32:57.627 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:32:58.746 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:32:58.763 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:32:58.763 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:32:58.763 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:32:58.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:32:58.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:32:59.202 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:32:59.253 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:32:59.255 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:32:59.297 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:33:00.376 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:35:00.451 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:35:00.492 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:35:01.591 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:35:01.607 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:35:01.607 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:35:01.607 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:35:01.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:35:01.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:35:02.045 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:35:02.095 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:35:02.097 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:35:02.136 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:35:03.182 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:37:03.246 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:37:03.285 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:37:04.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:37:04.420 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:37:04.420 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:37:04.420 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:37:04.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:37:04.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:37:04.861 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:37:04.902 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:37:04.904 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:37:04.949 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:37:05.999 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:39:06.064 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:39:06.109 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:39:07.171 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:39:07.188 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:39:07.188 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:39:07.188 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:39:07.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:39:07.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:39:07.658 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:39:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:39:07.711 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:39:07.754 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:39:08.826 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:41:08.891 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:41:08.932 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:41:09.960 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:41:09.977 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:41:09.977 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:41:09.977 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:41:09.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:41:10.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:41:10.421 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:41:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:41:10.462 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:41:10.501 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:41:11.548 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:43:11.640 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:43:11.679 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:43:12.746 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:43:12.765 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:43:12.765 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:43:12.765 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:43:12.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:43:12.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:43:13.215 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:43:13.259 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:43:13.260 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:43:13.301 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:43:14.350 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:45:14.420 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:45:14.470 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:45:15.504 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:45:15.522 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:45:15.522 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:45:15.522 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:45:15.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:45:15.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:45:15.965 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:45:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:45:16.009 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:45:16.047 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:45:17.113 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:47:17.203 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:47:17.244 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:47:18.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:47:18.357 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:47:18.357 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:47:18.357 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:47:18.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:47:18.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:47:18.793 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:47:18.832 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:47:18.834 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:47:18.870 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:47:19.887 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:49:19.956 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:49:19.995 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:49:21.044 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:49:21.062 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:49:21.062 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:49:21.062 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:49:21.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:49:21.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:49:21.504 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:49:21.550 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:49:21.552 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:49:21.591 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:49:22.625 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:51:22.692 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:51:22.733 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:51:23.795 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:51:23.812 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:51:23.812 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:51:23.813 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:51:23.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:51:24.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:51:24.255 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:51:24.302 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:51:24.304 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:51:24.343 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:51:25.414 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:53:25.476 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:53:25.531 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:53:26.593 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:53:26.610 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:53:26.610 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:53:26.610 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:53:26.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:53:26.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:53:27.052 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:53:27.092 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:53:27.093 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:53:27.132 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:53:28.172 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:55:28.235 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:55:28.297 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:55:29.436 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:55:29.453 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:55:29.453 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:55:29.453 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:55:29.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:55:29.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:55:29.891 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:55:29.938 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:55:29.939 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:55:29.984 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:55:31.090 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:57:31.153 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:57:31.201 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:57:32.263 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:57:32.280 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:57:32.280 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:57:32.281 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:57:32.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:57:32.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:57:32.726 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:57:32.776 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:57:32.777 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:57:32.818 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:57:33.949 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:59:34.020 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T20:59:34.125 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:59:35.212 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T20:59:35.230 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T20:59:35.230 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T20:59:35.230 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:59:35.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:59:35.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T20:59:35.659 INFO:teuthology.misc:Compressing logs... 2021-03-22T20:59:35.712 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T20:59:35.714 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T20:59:35.754 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T20:59:36.799 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:01:36.860 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:01:36.897 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:01:37.904 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:01:37.921 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:01:37.921 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:01:37.921 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:01:37.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:01:38.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:01:38.369 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:01:38.408 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:01:38.410 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:01:38.449 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:01:39.539 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:03:39.602 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:03:39.639 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:03:40.694 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:03:40.711 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:03:40.712 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:03:40.712 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:03:40.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:03:40.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:03:41.147 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:03:41.197 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:03:41.199 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:03:41.237 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:03:42.229 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:05:42.313 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:05:42.355 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:05:43.374 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:05:43.391 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:05:43.391 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:05:43.391 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:05:43.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:05:43.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:05:43.828 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:05:43.876 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:05:43.878 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:05:43.918 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:05:44.920 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:07:44.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:07:45.028 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:07:46.192 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:07:46.209 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:07:46.209 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:07:46.209 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:07:46.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:07:46.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:07:46.652 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:07:46.701 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:07:46.703 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:07:46.745 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:07:47.814 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:09:47.887 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:09:47.925 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:09:48.934 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:09:48.951 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:09:48.951 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:09:48.951 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:09:48.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:09:49.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:09:49.380 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:09:49.432 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:09:49.434 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:09:49.471 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:09:50.544 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:11:50.608 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:11:50.651 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:11:51.713 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:11:51.730 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:11:51.730 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:11:51.730 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:11:51.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:11:51.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:11:52.180 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:11:52.224 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:11:52.226 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:11:52.265 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:11:53.302 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:13:53.361 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:13:53.400 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:13:54.476 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:13:54.494 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:13:54.494 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:13:54.494 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:13:54.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:13:54.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:13:54.940 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:13:54.980 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:13:54.982 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:13:55.021 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:13:56.053 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:15:56.115 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:15:56.151 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:15:57.127 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:15:57.144 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:15:57.144 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:15:57.144 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:15:57.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:15:57.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:15:57.595 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:15:57.643 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:15:57.645 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:15:57.686 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:15:58.696 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:17:58.772 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:17:58.819 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:17:59.883 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:17:59.899 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:17:59.899 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:17:59.899 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:17:59.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:18:00.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:18:00.336 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:18:00.382 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:18:00.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:18:00.419 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:18:01.416 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:20:01.615 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:20:01.653 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:20:02.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:20:02.705 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:20:02.705 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:20:02.706 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:20:02.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:20:02.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:20:03.137 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:20:03.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:20:03.186 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:20:03.227 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:20:04.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:22:04.334 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:22:04.375 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:22:05.398 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:22:05.415 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:22:05.416 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:22:05.416 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:22:05.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:22:05.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:22:05.860 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:22:05.907 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:22:05.909 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:22:05.949 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:22:06.949 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:24:07.016 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:24:07.063 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:24:08.148 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:24:08.166 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:24:08.166 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:24:08.166 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:24:08.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:24:08.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:24:08.619 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:24:08.667 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:24:08.668 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:24:08.710 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:24:09.804 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:26:09.872 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:26:09.913 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:26:10.970 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:26:10.986 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:26:10.986 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:26:10.986 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:26:10.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:26:11.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:26:11.427 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:26:11.475 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:26:11.477 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:26:11.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:26:12.527 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:28:12.590 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:28:12.637 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:28:13.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:28:13.705 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:28:13.705 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:28:13.705 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:28:13.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:28:13.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:28:14.148 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:28:14.186 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:28:14.187 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:28:14.228 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:28:15.277 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:30:15.344 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:30:15.388 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:30:16.537 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:30:16.554 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:30:16.554 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:30:16.554 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:30:16.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:30:16.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:30:17.044 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:30:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:30:17.097 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:30:17.157 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:30:18.281 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:32:18.359 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:32:18.402 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:32:19.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:32:19.486 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:32:19.486 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:32:19.486 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:32:19.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:32:19.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:32:19.976 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:32:20.028 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:32:20.029 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:32:20.134 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:32:21.221 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:34:21.282 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:34:21.322 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:34:22.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:34:22.380 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:34:22.380 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:34:22.380 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:34:22.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:34:22.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:34:22.823 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:34:22.865 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:34:22.867 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:34:22.904 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:34:23.932 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:36:23.994 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:36:24.036 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:36:25.103 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:36:25.122 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:36:25.122 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:36:25.123 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:36:25.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:36:25.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:36:25.568 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:36:25.615 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:36:25.617 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:36:25.655 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:36:26.664 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:38:26.728 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:38:26.767 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:38:27.834 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:38:27.868 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:38:27.868 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:38:27.868 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:38:27.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:38:28.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:38:28.324 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:38:28.372 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:38:28.373 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:38:28.413 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:38:29.479 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:40:29.540 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:40:29.580 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:40:30.588 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:40:30.607 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:40:30.607 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:40:30.608 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:40:30.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:40:30.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:40:31.053 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:40:31.099 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:40:31.100 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:40:31.144 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:40:32.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:42:32.246 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:42:32.291 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:42:33.325 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:42:33.346 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:42:33.346 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:42:33.347 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:42:33.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:42:33.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:42:33.822 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:42:33.874 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:42:33.876 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:42:33.917 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:42:34.967 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:44:35.029 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:44:35.069 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:44:36.101 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:44:36.118 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:44:36.119 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:44:36.119 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:44:36.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:44:36.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:44:36.571 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:44:36.616 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:44:36.617 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:44:36.659 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:44:37.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:46:37.753 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:46:37.791 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:46:38.832 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:46:38.849 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:46:38.850 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:46:38.850 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:46:38.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:46:39.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:46:39.281 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:46:39.328 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:46:39.330 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:46:39.366 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:46:40.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:48:40.410 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:48:40.452 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:48:41.469 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:48:41.485 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:48:41.485 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:48:41.485 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:48:41.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:48:41.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:48:41.926 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:48:41.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:48:41.971 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:48:42.010 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:48:43.051 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:50:43.112 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:50:43.150 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:50:44.225 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:50:44.242 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:50:44.243 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:50:44.243 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:50:44.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:50:44.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:50:44.687 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:50:44.735 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:50:44.737 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:50:44.776 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:50:45.833 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:52:45.893 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:52:45.928 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:52:46.914 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:52:46.932 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:52:46.932 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:52:46.932 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:52:46.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:52:47.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:52:47.372 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:52:47.422 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:52:47.424 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:52:47.465 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:52:48.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:54:48.588 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:54:48.631 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:54:49.655 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:54:49.673 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:54:49.673 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:54:49.674 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:54:49.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:54:49.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:54:50.137 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:54:50.194 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:54:50.195 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:54:50.245 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:54:51.314 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:56:51.374 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:56:51.417 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:56:52.486 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:56:52.503 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:56:52.503 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:56:52.503 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:56:52.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:56:52.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:56:53.011 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:56:53.064 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:56:53.066 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:56:53.107 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:56:54.175 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:58:54.236 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T21:58:54.276 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:58:55.266 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T21:58:55.282 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T21:58:55.282 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T21:58:55.283 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:58:55.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:58:55.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T21:58:55.717 INFO:teuthology.misc:Compressing logs... 2021-03-22T21:58:55.766 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T21:58:55.768 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T21:58:55.806 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T21:58:56.800 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:00:56.861 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:00:56.903 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:00:57.988 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:00:58.005 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:00:58.006 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:00:58.006 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:00:58.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:00:58.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:00:58.456 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:00:58.507 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:00:58.508 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:00:58.547 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:00:59.634 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:02:59.696 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:02:59.737 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:03:00.776 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:03:00.794 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:03:00.794 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:03:00.794 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:03:00.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:03:00.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:03:01.239 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:03:01.283 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:03:01.284 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:03:01.323 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:03:02.350 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:05:02.472 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:05:02.511 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:05:03.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:05:03.570 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:05:03.570 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:05:03.571 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:05:03.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:05:03.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:05:04.007 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:05:04.054 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:05:04.056 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:05:04.095 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:05:05.076 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:07:05.137 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:07:05.177 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:07:06.226 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:07:06.311 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:07:06.311 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:07:06.311 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:07:06.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:07:06.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:07:06.744 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:07:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:07:06.792 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:07:06.830 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:07:07.927 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:09:07.995 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:09:08.036 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:09:09.112 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:09:09.129 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:09:09.129 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:09:09.129 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:09:09.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:09:09.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:09:09.568 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:09:09.615 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:09:09.617 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:09:09.658 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:09:10.725 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:11:10.785 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:11:10.822 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:11:11.872 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:11:11.889 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:11:11.889 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:11:11.889 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:11:11.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:11:12.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:11:12.331 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:11:12.378 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:11:12.380 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:11:12.422 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:11:13.475 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:13:13.538 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:13:13.579 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:13:14.630 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:13:14.652 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:13:14.652 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:13:14.652 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:13:14.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:13:14.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:13:15.094 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:13:15.142 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:13:15.144 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:13:15.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:13:16.240 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:15:16.302 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:15:16.344 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:15:17.410 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:15:17.427 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:15:17.427 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:15:17.427 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:15:17.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:15:17.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:15:17.872 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:15:17.918 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:15:17.920 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:15:17.963 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:15:18.987 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:17:19.050 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:17:19.090 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:17:20.897 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:17:20.914 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:17:20.915 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:17:20.915 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:17:20.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:17:21.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:17:21.355 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:17:21.403 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:17:21.404 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:17:21.444 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:17:22.511 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:19:22.574 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:19:22.626 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:19:24.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:19:24.563 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:19:24.563 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:19:24.563 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:19:24.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:19:24.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:19:25.007 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:19:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:19:25.048 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:19:25.087 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:19:26.297 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:21:26.365 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:21:26.404 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:21:28.183 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:21:28.201 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:21:28.201 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:21:28.201 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:21:28.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:21:28.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:21:28.634 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:21:28.685 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:21:28.686 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:21:28.727 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:21:29.771 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:23:29.830 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:23:29.869 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:23:31.773 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:23:31.790 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:23:31.790 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:23:31.790 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:23:31.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:23:31.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:23:32.234 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:23:32.282 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T22:23:32.283 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:23:32.323 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:23:33.381 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:25:33.446 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:25:33.487 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:25:35.372 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:25:35.414 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:25:35.415 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:25:35.415 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:25:35.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:26:35.418 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T22:26:35.461 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:26:36.565 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:28:36.630 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:28:36.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:28:37.729 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:28:37.747 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:28:37.748 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:28:37.748 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:28:37.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:28:40.802 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T22:28:40.840 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:28:41.849 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:30:41.914 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:30:41.952 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:30:42.952 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:30:42.969 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:30:42.969 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:30:42.969 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:30:42.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:30:43.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:30:43.295 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T22:31:17.955 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T22:31:17.994 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:31:19.098 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:33:19.165 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:33:19.203 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:33:20.273 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:33:20.290 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:33:20.290 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:33:20.290 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:33:20.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:33:20.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:33:20.532 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T22:33:20.565 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:33:21.662 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:33:22.729 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:33:23.832 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:33:24.893 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:33:25.990 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:33:27.110 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:33:28.206 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:33:29.306 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:33:30.408 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:33:30.409 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T22:33:30.447 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:33:31.508 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:35:31.575 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:35:31.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:35:32.836 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:35:32.857 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:35:32.858 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:35:32.858 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:35:32.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:36:05.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:36:05.320 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T22:36:05.467 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T22:36:05.703 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:36:05.703 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:36:05.738 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:36:05.738 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:36:05.739 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:36:05.767 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T22:36:05.802 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T22:36:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T22:36:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:36:05.853 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T22:36:05.854 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:36:05.856 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:36:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T22:36:05.915 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T22:36:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T22:36:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:36:06.139 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T22:36:06.140 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:36:06.189 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:36:07.487 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:38:07.558 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:38:07.727 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:38:11.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:38:11.341 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:38:11.342 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:38:11.342 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:38:11.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:38:11.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:38:11.896 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T22:39:11.812 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T22:39:11.955 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:39:16.329 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:41:16.612 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:41:16.738 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:41:19.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:41:19.563 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:41:19.563 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:41:19.563 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:41:19.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:41:19.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:41:19.891 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T22:41:20.115 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T22:41:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:41:20.148 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:41:20.171 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:41:20.171 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:41:20.172 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:41:20.385 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:41:20.413 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T22:41:20.432 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T22:41:20.521 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T22:41:20.521 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:41:20.537 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T22:41:20.537 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:41:20.667 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:41:23.109 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:43:23.222 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:43:23.263 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:43:24.371 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:43:24.390 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:43:24.390 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:43:24.390 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:43:24.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:43:24.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:43:24.653 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T22:43:24.797 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T22:43:24.827 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:43:24.827 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:43:24.843 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:43:24.843 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:43:24.845 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:43:25.236 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:43:25.260 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T22:43:25.383 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T22:43:25.384 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:43:25.426 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:43:26.518 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:45:26.583 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:45:26.707 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:45:27.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:45:27.792 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:45:27.793 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:45:27.793 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:45:27.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:45:28.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:45:28.056 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T22:45:28.220 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T22:45:28.252 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:45:28.253 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:45:28.284 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:45:28.284 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:45:28.286 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:45:28.647 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:45:28.670 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T22:45:28.680 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T22:45:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/fe8de7ea-8b5f-11eb-90a2-001a4aab830c/ceph-mon.a.log: file size changed while zipping 2021-03-22T22:45:28.729 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:45:28.769 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:45:29.808 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:47:29.871 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:47:29.912 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:47:31.018 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:47:31.038 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:47:31.038 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:47:31.038 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:47:31.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:47:31.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:47:31.298 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T22:47:31.440 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T22:47:31.470 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:47:31.470 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:47:31.488 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:47:31.489 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:47:31.490 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:47:31.874 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:47:31.905 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T22:47:31.936 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/fe8de7ea-8b5f-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T22:47:31.946 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:47:31.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:47:33.069 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:49:33.138 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:49:33.178 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:49:34.236 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:49:34.254 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:49:34.254 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:49:34.254 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:49:34.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:49:34.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:49:34.530 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T22:49:34.688 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T22:49:34.721 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:49:34.721 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:49:34.747 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:49:34.747 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:49:34.750 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:49:35.178 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:49:35.206 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T22:49:35.224 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T22:49:35.325 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/fe8de7ea-8b5f-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T22:49:35.386 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:49:35.425 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:49:36.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:51:36.564 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:51:36.608 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:51:37.686 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:51:37.704 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:51:37.704 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:51:37.704 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:51:37.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:51:37.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:51:37.969 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T22:51:38.114 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T22:51:38.155 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:51:38.155 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:51:38.177 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T22:51:38.177 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T22:51:38.180 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:51:38.537 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:51:38.570 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-22T22:51:38.570 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/fe8de7ea-8b5f-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-22T22:51:38.571 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T22:51:38.609 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:51:39.653 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:53:39.717 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:53:39.759 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:53:40.790 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:53:40.806 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:53:40.806 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:53:40.806 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:53:40.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:54:12.516 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:54:13.613 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:54:14.674 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:54:15.776 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:54:16.832 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:54:17.933 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:54:19.033 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:54:20.093 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:54:21.192 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:54:22.254 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T22:54:22.255 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T22:54:22.295 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:54:23.334 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:56:23.398 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:56:23.439 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:56:24.496 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:56:24.513 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:56:24.514 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:56:24.514 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:56:24.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:56:39.875 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T22:56:39.911 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:56:40.933 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:58:41.042 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T22:58:41.082 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:58:42.122 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T22:58:42.139 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T22:58:42.140 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T22:58:42.140 INFO:teuthology.misc:Compressing logs... 2021-03-22T22:58:42.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T22:59:42.142 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T22:59:42.181 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T22:59:43.233 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:01:43.302 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:01:43.345 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:01:44.434 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:01:44.451 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:01:44.451 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:01:44.451 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:01:44.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:01:44.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:01:44.820 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T23:01:45.120 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T23:01:45.140 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:01:45.140 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:01:45.182 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:01:45.182 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:01:45.184 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:01:45.240 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T23:01:45.323 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T23:01:45.323 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:01:45.325 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:01:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T23:01:45.380 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T23:01:45.400 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T23:01:45.400 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:01:45.440 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T23:01:45.440 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:01:45.486 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:01:46.585 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:03:46.652 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:03:46.697 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:03:47.849 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:03:47.868 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:03:47.868 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:03:47.868 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:03:47.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:03:48.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:03:51.333 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T23:03:51.394 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:03:52.727 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:05:52.796 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:05:52.838 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:05:54.057 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:05:54.075 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:05:54.075 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:05:54.075 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:05:54.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:05:54.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:05:54.726 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T23:05:54.784 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:05:54.784 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:05:54.787 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:05:54.932 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:05:54.983 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T23:05:55.045 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T23:05:55.045 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:05:55.085 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:05:56.176 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:07:56.244 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:07:56.302 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:07:57.430 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:07:57.447 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:07:57.448 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:07:57.448 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:07:57.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:07:57.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:07:58.128 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T23:07:58.187 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:07:58.188 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:07:58.190 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:07:58.331 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:07:58.388 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T23:07:58.444 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T23:07:58.444 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:07:58.489 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:07:59.612 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:09:59.678 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:09:59.720 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:10:00.802 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:10:00.819 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:10:00.819 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:10:00.820 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:10:00.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:10:01.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:10:01.576 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:10:01.722 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:10:01.924 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:10:02.995 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:12:03.059 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:12:03.113 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:12:04.307 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:12:04.327 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:12:04.327 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:12:04.327 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:12:04.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:12:04.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:12:05.070 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:12:05.213 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:12:06.749 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping 2021-03-22T23:12:13.946 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2021-03-22T23:12:15.469 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2021-03-22T23:12:15.470 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-22T23:12:15.991 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T23:12:16.042 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:12:17.190 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:14:17.259 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:14:17.299 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:14:18.476 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:14:18.494 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:14:18.495 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:14:18.495 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:14:18.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:14:18.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:14:19.252 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:14:19.404 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:14:19.460 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists; not overwritten 2021-03-22T23:14:19.461 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T23:14:19.461 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T23:14:19.461 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-22T23:14:19.461 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.7.log.gz already exists; not overwritten 2021-03-22T23:14:19.461 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-22T23:14:19.465 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T23:14:19.509 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:14:20.623 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:16:20.684 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:16:20.729 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:16:21.773 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:16:21.790 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:16:21.791 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:16:21.791 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:16:21.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:16:24.866 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T23:16:24.906 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:16:26.017 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:18:26.085 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:18:26.126 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:18:27.212 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:18:27.234 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:18:27.234 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:18:27.234 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:18:27.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:18:28.070 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T23:18:28.110 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:18:29.185 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:20:29.247 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:20:29.285 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:20:30.389 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:20:30.408 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:20:30.408 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:20:30.408 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:20:30.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:20:30.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:20:30.760 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T23:21:02.205 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:21:03.307 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:21:04.366 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:21:05.463 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:21:06.521 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:21:07.623 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:21:08.721 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:21:09.780 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:21:10.883 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:21:11.944 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:21:11.945 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T23:21:12.039 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:21:13.075 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:23:13.139 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:23:13.180 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:23:14.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:23:14.257 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:23:14.258 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:23:14.258 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:23:14.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:23:14.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:23:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T23:23:30.561 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T23:23:30.591 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:23:30.591 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:23:30.627 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:23:30.628 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:23:30.629 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:23:30.694 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T23:23:30.771 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T23:23:30.772 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:23:30.773 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:23:30.796 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T23:23:30.837 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T23:23:30.857 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T23:23:30.858 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:23:30.903 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T23:23:30.903 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:23:30.952 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:23:32.043 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:25:32.107 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:25:32.151 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:25:33.287 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:25:33.304 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:25:33.305 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:25:33.305 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:25:33.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:25:33.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:25:33.648 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T23:25:33.857 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T23:25:33.874 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:25:33.874 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:25:33.922 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:25:33.922 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:25:33.923 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:25:33.987 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T23:25:34.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T23:25:34.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:25:34.070 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:25:34.080 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-22T23:25:34.137 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T23:25:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T23:25:34.166 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:25:34.199 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T23:25:34.200 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:25:34.239 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:25:35.311 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:27:35.376 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:27:35.418 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:27:36.549 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:27:36.567 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:27:36.567 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:27:36.568 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:27:36.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:27:36.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:27:37.648 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T23:27:37.705 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:27:37.705 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:27:37.707 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:27:37.770 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T23:27:37.846 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T23:27:37.847 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:27:37.848 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:27:37.912 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-22T23:27:37.972 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2021-03-22T23:27:38.090 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-22T23:27:38.090 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:27:38.139 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:27:39.213 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:29:39.302 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:29:39.341 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:29:40.393 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:29:40.411 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:29:40.411 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:29:40.411 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:29:40.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:29:41.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:29:41.319 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-22T23:29:41.392 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:29:41.393 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:29:41.394 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:29:41.617 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:29:41.662 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-22T23:29:41.663 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-22T23:29:41.666 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T23:29:41.704 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:29:42.746 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:31:42.820 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:31:42.861 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:31:43.982 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:31:43.999 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:31:43.999 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:31:43.999 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:31:43.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:32:02.371 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T23:32:02.411 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:32:03.518 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:34:03.613 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:34:03.656 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:34:04.769 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:34:04.786 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:34:04.787 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:34:04.787 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:34:04.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:34:04.867 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:34:05.927 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:34:07.029 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:34:08.088 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:34:09.189 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:34:10.290 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:34:11.354 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:34:12.454 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:34:13.552 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:34:14.614 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:34:14.614 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T23:34:14.654 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:34:15.689 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:36:15.754 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:36:15.807 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:36:16.985 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:36:17.006 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:36:17.006 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:36:17.006 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:36:17.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:36:32.456 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-22T23:36:32.456 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T23:36:32.496 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:36:33.577 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:38:33.641 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:38:33.684 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:38:34.747 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:38:34.764 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:38:34.765 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:38:34.765 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:38:34.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:38:35.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:38:35.125 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T23:38:35.361 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:38:35.361 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:38:35.417 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:38:35.435 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-22T23:38:35.485 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T23:38:35.485 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T23:38:35.528 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:38:36.631 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:40:36.694 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:40:36.753 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:40:37.934 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:40:37.951 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:40:37.951 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:40:37.951 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:40:37.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:41:37.954 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T23:41:37.997 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:41:39.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:43:39.193 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:43:39.237 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:43:40.382 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:43:40.402 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:43:40.402 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:43:40.402 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:43:40.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:43:40.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:43:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T23:43:41.177 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:43:41.177 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:43:41.234 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:43:41.311 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T23:43:41.313 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T23:43:41.356 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:43:42.447 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:45:42.513 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:45:42.553 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:45:43.603 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:45:43.623 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:45:43.624 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:45:43.624 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:45:43.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:45:43.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:45:43.961 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-22T23:45:44.216 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-22T23:45:44.216 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:45:44.269 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:45:44.337 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T23:45:44.338 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T23:45:44.377 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:45:45.452 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:47:45.514 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:47:45.557 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:47:46.656 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:47:46.673 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:47:46.673 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:47:46.673 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:47:46.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:47:47.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:47:47.438 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:47:47.504 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T23:47:47.506 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T23:47:47.553 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:47:48.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:49:48.678 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:49:48.719 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:49:49.809 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:49:49.826 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:49:49.826 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:49:49.827 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:49:49.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:49:50.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:49:50.502 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:49:50.570 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T23:49:50.572 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T23:49:50.616 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:49:51.684 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:51:51.747 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:51:51.786 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:51:52.896 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:51:52.918 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:51:52.918 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:51:52.918 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:51:52.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:51:53.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:51:53.593 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:51:53.660 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-22T23:51:53.662 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-22T23:51:53.700 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:51:54.769 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:53:54.833 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:53:54.875 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:53:55.964 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:53:55.984 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:53:55.984 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:53:55.984 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:53:55.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:53:56.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:53:56.635 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:53:56.699 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-22T23:53:56.764 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-22T23:53:56.764 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-22T23:53:56.766 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:53:57.042 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/fe6fa9b6-8b68-11eb-90a2-001a4aab830c/ceph-mon.a.log: file size changed while zipping 2021-03-22T23:53:58.385 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:53:59.491 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:55:59.554 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:55:59.597 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:56:00.737 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:56:00.753 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:56:00.753 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:56:00.753 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:56:00.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:56:01.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:56:04.162 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T23:56:04.206 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:56:05.307 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:58:05.374 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-22T23:58:05.414 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:58:06.479 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-22T23:58:06.496 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-22T23:58:06.496 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-22T23:58:06.496 INFO:teuthology.misc:Compressing logs... 2021-03-22T23:58:06.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:58:06.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T23:58:06.927 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:58:07.985 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:58:09.088 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:58:10.186 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:58:11.249 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:58:12.425 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:58:13.483 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:58:14.586 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:58:15.692 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:58:16.750 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-22T23:58:16.751 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-22T23:58:16.799 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-22T23:58:17.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:00:17.970 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:00:18.014 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:00:19.175 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:00:19.191 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:00:19.192 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:00:19.192 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:00:19.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:00:19.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:00:51.139 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T00:00:51.188 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:00:52.212 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:02:52.274 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:02:52.317 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:02:53.414 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:02:53.432 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:02:53.432 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:02:53.432 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:02:53.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:02:53.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:02:53.979 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T00:02:54.053 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:02:54.053 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:02:54.056 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:02:54.083 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-23T00:02:54.222 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T00:02:54.223 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:02:54.224 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:02:54.241 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T00:02:54.266 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:02:54.307 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:02:55.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:04:55.436 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:04:55.478 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:04:56.521 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:04:56.548 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:04:56.548 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:04:56.548 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:04:56.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:04:56.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:05:29.117 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T00:05:29.178 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:05:29.179 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:05:29.181 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:05:29.239 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-23T00:05:29.320 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T00:05:29.321 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:05:29.322 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:05:29.339 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T00:05:29.368 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:05:29.413 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:05:30.494 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:07:30.585 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:07:30.635 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:07:31.821 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:07:31.840 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:07:31.841 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:07:31.841 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:07:31.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:07:32.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:07:32.391 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T00:07:32.428 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:07:32.428 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:07:32.430 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:07:32.594 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:07:32.613 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T00:07:32.625 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:07:32.668 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:07:33.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:09:33.822 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:09:33.862 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:09:35.827 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:09:35.845 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:09:35.846 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:09:35.846 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:09:35.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:09:36.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:09:36.383 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T00:09:36.420 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:09:36.420 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:09:36.423 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:09:36.462 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-23T00:09:36.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T00:09:36.520 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:09:36.570 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:09:36.608 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T00:09:36.609 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/log/ceph': No such file or directory 2021-03-23T00:09:36.610 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:09:36.652 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:09:37.802 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:11:37.868 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:11:37.913 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:11:39.865 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:11:39.882 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:11:39.882 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:11:39.882 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:11:39.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:11:42.947 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T00:11:42.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:11:44.118 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:13:44.183 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:13:44.228 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:13:45.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:13:45.368 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:13:45.368 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:13:45.368 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:13:45.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:14:17.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:14:17.493 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T00:14:17.645 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:14:17.645 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:14:17.666 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:14:17.712 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T00:14:17.781 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T00:14:17.781 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:14:17.824 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:14:17.857 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T00:14:17.923 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-23T00:14:17.924 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:14:18.001 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:14:19.129 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:16:19.202 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:16:19.248 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:16:20.388 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:16:20.406 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:16:20.407 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:16:20.407 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:16:20.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:17:20.410 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T00:17:20.477 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:17:22.079 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:19:22.146 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:19:22.189 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:19:23.359 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:19:23.376 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:19:23.376 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:19:23.376 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:19:23.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:19:23.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:19:23.648 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T00:19:23.836 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:19:23.836 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:19:23.852 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:19:24.004 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:19:24.034 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T00:19:24.215 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-23T00:19:24.216 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:19:24.351 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:19:25.552 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:21:25.616 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:21:25.662 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:21:26.850 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:21:26.867 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:21:26.867 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:21:26.867 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:21:26.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:21:27.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:21:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T00:21:27.397 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:21:27.397 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:21:27.423 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:21:27.465 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:21:27.468 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:21:27.517 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:21:28.704 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:23:28.842 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:23:28.886 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:23:30.006 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:23:30.023 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:23:30.023 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:23:30.023 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:23:30.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:23:30.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:23:30.498 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:23:30.545 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:23:30.548 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:23:30.591 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:23:31.727 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:25:31.790 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:25:31.830 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:25:32.896 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:25:32.914 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:25:32.915 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:25:32.915 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:25:32.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:25:33.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:25:33.380 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:25:33.429 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:25:33.431 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:25:33.469 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:25:34.550 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:27:34.639 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:27:34.686 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:27:35.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:27:35.873 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:27:35.873 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:27:35.874 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:27:35.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:27:54.434 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T00:27:54.479 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:27:55.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:29:55.704 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:29:55.743 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:29:56.976 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:29:56.993 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:29:56.993 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:29:56.993 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:29:56.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:30:00.066 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T00:30:00.105 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:30:01.210 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:32:01.296 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:32:01.340 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:32:02.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:32:02.419 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:32:02.419 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:32:02.419 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:32:02.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:32:04.674 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T00:32:04.714 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:32:05.768 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:34:05.835 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:34:05.907 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:34:07.132 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:34:07.159 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:34:07.159 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:34:07.159 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:34:07.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:34:07.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:34:07.429 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T00:34:07.762 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T00:34:07.796 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:34:07.797 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:34:07.837 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:34:07.837 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:34:07.839 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:34:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-23T00:34:07.913 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T00:34:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T00:34:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:34:07.980 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T00:34:07.980 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:34:07.984 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:34:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-23T00:34:08.048 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T00:34:08.066 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-23T00:34:08.066 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:34:08.114 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-23T00:34:08.114 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:34:08.157 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:34:09.375 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:36:09.441 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:36:09.509 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:36:10.765 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:36:10.786 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:36:10.786 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:36:10.786 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:36:10.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:36:43.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:36:43.164 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T00:37:17.638 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T00:37:17.754 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:37:18.841 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:39:18.912 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:39:18.960 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:39:20.141 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:39:20.157 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:39:20.157 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:39:20.158 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:39:20.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:39:20.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:39:20.402 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T00:39:20.630 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T00:39:20.650 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:39:20.650 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:39:20.713 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:39:20.713 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:39:20.714 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:39:20.869 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:39:20.891 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-23T00:39:20.921 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T00:39:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-23T00:39:20.940 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:39:20.988 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-23T00:39:20.989 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:39:21.030 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:39:22.105 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:41:22.171 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:41:22.214 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:41:23.271 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:41:23.291 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:41:23.291 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:41:23.291 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:41:23.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:41:23.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:41:23.505 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T00:41:23.764 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T00:41:23.784 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:41:23.784 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:41:23.830 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:41:23.831 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:41:23.832 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:41:23.899 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:41:23.900 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:41:23.940 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:41:25.000 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:43:25.066 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:43:25.105 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:43:26.146 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:43:26.163 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:43:26.163 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:43:26.163 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:43:26.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:43:26.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:43:26.588 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T00:43:26.657 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T00:43:26.657 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T00:43:26.658 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:43:26.726 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:43:26.728 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:43:26.771 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:43:27.834 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:45:27.900 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:45:27.952 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:45:29.039 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:45:29.061 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:45:29.061 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:45:29.061 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:45:29.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:45:29.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:45:29.559 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:45:29.626 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:45:29.627 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:45:29.666 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:45:30.726 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:47:30.799 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:47:30.853 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:47:32.021 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:47:32.039 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:47:32.039 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:47:32.040 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:47:32.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:47:32.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:47:32.600 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:47:32.671 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:47:32.671 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:47:32.712 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:47:33.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:49:33.815 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:49:33.883 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:49:34.975 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:49:34.994 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:49:34.994 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:49:34.994 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:49:34.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:49:35.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:49:35.507 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:49:35.576 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:49:35.576 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:49:35.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:49:36.728 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:51:36.801 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:51:36.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:51:37.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:51:37.920 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:51:37.920 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:51:37.920 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:51:37.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:51:38.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:51:38.478 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:51:38.543 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:51:38.545 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:51:38.585 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:51:39.722 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:53:39.794 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:53:39.838 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:53:40.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:53:40.956 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:53:40.956 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:53:40.956 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:53:40.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:53:41.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:53:41.459 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:53:41.525 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:53:41.527 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:53:41.567 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:53:42.617 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:55:42.687 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:55:42.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:55:43.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:55:43.808 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:55:43.808 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:55:43.809 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:55:43.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:55:43.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:55:44.315 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:55:44.383 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:55:44.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:55:44.425 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:55:45.551 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:57:45.625 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:57:45.664 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:57:46.731 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:57:46.749 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:57:46.749 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:57:46.749 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:57:46.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:57:46.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:57:47.261 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:57:47.285 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-23T00:57:47.328 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T00:57:47.329 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T00:57:47.375 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:57:48.458 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:59:48.524 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T00:59:48.572 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T00:59:49.650 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T00:59:49.667 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T00:59:49.667 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T00:59:49.668 INFO:teuthology.misc:Compressing logs... 2021-03-23T00:59:49.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T00:59:49.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:00:08.515 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T01:00:08.561 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:00:09.689 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:02:09.772 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:02:09.817 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:02:10.910 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:02:10.927 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:02:10.927 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:02:10.927 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:02:10.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:02:11.064 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T01:02:12.121 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T01:02:13.222 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T01:02:14.287 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T01:02:15.385 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T01:02:16.485 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T01:02:17.606 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T01:02:19.005 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T01:02:20.068 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T01:02:21.175 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T01:02:21.175 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T01:02:21.215 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:02:22.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:04:22.415 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:04:22.459 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:04:23.649 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:04:23.669 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:04:23.670 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:04:23.670 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:04:23.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:04:25.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:04:25.219 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:04:57.463 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:04:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:04:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:04:57.615 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:04:57.615 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:04:57.616 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:04:57.643 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T01:04:57.647 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T01:04:57.717 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T01:04:57.717 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:04:57.742 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T01:04:57.742 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:04:57.743 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:04:57.786 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T01:04:57.807 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T01:04:57.857 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-23T01:04:57.857 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:04:57.881 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-23T01:04:57.882 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:04:57.930 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:04:59.120 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:06:59.216 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:06:59.266 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:07:00.466 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:07:00.483 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:07:00.483 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:07:00.483 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:07:00.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:07:00.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:07:00.737 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:07:00.963 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:07:01.001 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:07:01.002 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:07:01.030 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:07:01.030 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:07:01.032 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:07:01.200 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:07:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T01:07:01.253 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T01:07:01.305 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-23T01:07:01.305 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:07:01.332 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-23T01:07:01.333 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:07:01.387 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:07:02.634 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:09:02.698 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:09:02.738 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:09:03.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:09:03.857 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:09:03.857 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:09:03.857 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:09:03.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:09:04.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:09:04.110 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:09:04.319 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:09:04.347 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:09:04.348 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:09:04.387 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:09:04.387 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:09:04.388 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:09:04.453 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:09:04.455 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:09:04.499 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:09:05.649 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:11:05.722 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:11:05.762 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:11:06.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:11:06.843 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:11:06.843 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:11:06.843 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:11:06.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:11:07.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:11:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:11:07.348 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:11:07.380 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:11:07.380 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:11:07.414 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:11:07.414 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:11:07.415 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:11:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:11:07.458 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:11:07.501 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:11:08.531 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:13:08.598 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:13:08.712 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:13:10.671 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:13:10.758 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:13:10.758 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:13:10.758 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:13:10.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:13:10.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:13:11.294 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:13:11.325 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:13:11.337 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:13:11.379 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:13:12.528 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:15:12.590 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:15:12.633 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:15:13.800 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:15:13.821 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:15:13.822 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:15:13.822 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:15:13.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:15:14.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:15:14.380 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:15:14.411 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:15:14.423 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:15:14.472 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:15:15.545 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:17:15.622 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:17:15.662 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:17:17.865 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:17:17.882 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:17:17.882 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:17:17.882 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:17:17.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:17:18.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:17:18.431 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:17:18.457 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:17:18.473 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:17:18.523 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:17:20.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:19:21.021 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:19:21.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:19:23.327 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:19:23.344 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:19:23.345 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:19:23.345 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:19:23.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:19:23.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:19:23.887 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:19:23.918 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:19:23.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:19:23.976 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:19:25.115 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:21:25.180 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:21:25.224 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:21:26.455 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:21:26.472 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:21:26.472 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:21:26.473 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:21:26.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:22:01.031 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T01:22:01.091 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:22:02.208 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:24:02.277 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:24:02.317 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:24:04.327 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:24:04.353 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:24:04.353 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:24:04.353 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:24:04.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:24:07.426 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T01:24:07.467 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:24:09.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:26:09.117 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:26:09.161 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:26:10.222 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:26:10.239 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:26:10.240 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:26:10.240 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:26:10.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:26:10.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:26:10.570 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:26:10.880 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:26:10.882 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:26:10.911 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:26:10.938 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T01:26:10.978 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:26:10.980 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:26:11.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:26:12.133 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:28:12.202 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:28:12.246 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:28:14.149 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:28:14.167 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:28:14.167 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:28:14.167 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:28:14.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:28:14.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:28:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:28:14.693 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:28:14.694 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:28:14.724 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:28:14.757 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T01:28:14.791 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:28:14.792 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:28:14.843 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:28:16.007 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:30:16.073 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:30:16.114 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:30:17.321 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:30:17.340 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:30:17.340 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:30:17.340 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:30:17.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:30:17.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:30:17.612 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:30:17.852 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:30:17.852 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:30:17.884 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:30:17.949 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:30:17.954 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:30:18.003 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:30:19.158 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:32:19.243 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:32:19.289 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:32:20.413 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:32:20.430 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:32:20.431 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:32:20.431 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:32:20.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:32:20.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:32:20.685 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:32:20.920 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:32:20.921 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:32:20.961 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:32:21.026 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:32:21.027 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:32:21.073 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:32:22.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:34:22.268 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:34:22.310 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:34:23.559 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:34:23.582 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:34:23.583 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:34:23.583 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:34:23.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:34:23.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:34:23.857 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:34:24.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:34:24.183 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:34:24.218 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:34:24.271 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:34:24.273 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:34:24.320 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:34:25.661 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:36:25.742 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:36:25.814 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:36:27.785 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:36:27.803 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:36:27.803 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:36:27.803 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:36:27.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:36:28.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:36:28.339 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:36:28.370 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:36:28.381 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:36:28.436 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:36:29.611 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:38:29.674 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:38:29.719 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:38:30.777 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:38:30.795 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:38:30.795 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:38:30.795 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:38:30.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:38:31.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:38:31.353 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:38:31.384 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:38:31.396 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:38:31.439 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:38:32.483 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:40:32.549 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:40:32.603 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:40:33.686 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:40:33.704 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:40:33.704 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:40:33.704 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:40:33.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:40:33.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:40:37.026 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T01:40:37.072 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:40:38.112 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:42:38.174 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:42:38.219 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:42:39.374 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:42:39.391 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:42:39.391 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:42:39.391 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:42:39.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:42:39.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:42:42.690 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T01:42:42.730 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:42:43.849 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:44:43.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:44:43.994 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:44:45.172 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:44:45.191 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:44:45.192 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:44:45.192 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:44:45.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:44:45.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:44:48.486 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T01:44:48.529 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:44:49.669 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:46:49.740 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:46:49.784 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:46:50.998 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:46:51.016 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:46:51.016 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:46:51.017 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:46:51.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:46:51.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:46:51.659 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T01:46:51.733 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:46:51.734 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:46:51.735 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:46:51.783 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:46:51.787 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:46:51.831 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:46:52.933 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:48:52.998 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:48:53.122 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:48:55.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:48:55.076 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:48:55.076 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:48:55.077 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:48:55.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:48:55.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:48:55.641 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T01:48:55.700 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:48:55.700 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:48:55.703 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:48:55.755 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:48:55.757 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:48:55.800 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:48:57.859 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:50:57.922 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:50:57.965 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:50:59.067 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:50:59.085 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:50:59.085 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:50:59.085 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:50:59.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:50:59.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:51:02.374 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T01:51:02.418 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:51:03.547 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:53:03.620 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:53:03.663 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:53:04.786 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:53:04.805 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:53:04.806 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:53:04.806 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:53:04.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:53:05.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:53:05.369 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T01:53:05.429 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:53:05.429 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:53:05.432 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:53:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T01:53:05.475 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T01:53:05.518 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:53:06.665 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:55:06.735 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:55:06.783 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:55:07.933 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:55:07.956 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:55:07.956 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:55:07.956 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:55:07.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:55:11.014 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T01:55:11.061 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:55:12.157 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:57:12.240 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:57:12.288 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:57:13.384 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:57:13.402 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:57:13.402 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:57:13.402 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:57:13.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:57:16.486 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T01:57:16.526 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:57:17.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:59:17.670 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T01:59:17.721 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:59:18.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T01:59:18.793 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T01:59:18.793 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T01:59:18.794 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:59:18.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:59:50.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T01:59:51.067 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T01:59:51.310 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T01:59:51.365 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:59:51.365 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:59:51.381 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T01:59:51.381 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:59:51.384 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:59:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T01:59:51.480 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T01:59:51.481 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:59:51.516 INFO:teuthology.misc:Compressing logs... 2021-03-23T01:59:51.546 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T01:59:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2021-03-23T01:59:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T01:59:51.720 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T01:59:52.787 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:01:52.855 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:01:52.902 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:01:53.963 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:01:53.981 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:01:53.982 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:01:53.982 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:01:53.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:01:54.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:01:54.234 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T02:01:54.544 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:01:54.571 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:01:54.572 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:01:54.605 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:01:54.605 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:01:54.610 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:01:54.783 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:01:54.808 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T02:01:54.838 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T02:01:54.839 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:01:54.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:01:56.001 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:03:56.070 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:03:56.122 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:03:57.229 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:03:57.247 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:03:57.247 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:03:57.247 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:03:57.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:03:57.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:03:57.510 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T02:03:57.811 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:03:57.839 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:03:57.840 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:03:57.871 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:03:57.871 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:03:57.874 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:03:57.911 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T02:03:57.918 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:03:57.959 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:03:59.060 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:05:59.126 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:05:59.169 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:06:00.292 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:06:00.310 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:06:00.310 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:06:00.310 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:06:00.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:06:00.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:06:00.560 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T02:06:00.873 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:06:00.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:06:00.903 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:06:00.936 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:06:00.936 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:06:00.940 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:06:00.970 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T02:06:00.984 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:06:01.042 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:06:02.174 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:08:02.242 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:08:02.291 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:08:03.593 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:08:03.614 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:08:03.616 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:08:03.617 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:08:03.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:08:03.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:08:04.194 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:08:04.258 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:08:04.258 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:08:04.261 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:08:04.291 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T02:08:04.304 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:08:04.349 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:08:05.535 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:10:05.607 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:10:05.650 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:10:06.784 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:10:06.808 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:10:06.808 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:10:06.809 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:10:06.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:10:07.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:10:07.398 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:10:07.458 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:10:07.458 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:10:07.460 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:10:07.498 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T02:10:07.505 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:10:07.548 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:10:08.622 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:12:08.686 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:12:08.729 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:12:09.818 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:12:09.835 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:12:09.835 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:12:09.835 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:12:09.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:12:10.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:12:10.393 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:12:10.455 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:12:10.455 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:12:10.459 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:12:10.502 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T02:12:10.503 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:12:10.545 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:12:11.652 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:14:11.719 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:14:11.770 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:14:12.954 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:14:12.970 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:14:12.971 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:14:12.971 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:14:12.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:14:16.034 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:14:16.080 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:14:17.198 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:16:17.262 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:16:17.304 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:16:18.382 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:16:18.399 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:16:18.399 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:16:18.399 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:16:18.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:16:18.487 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:16:19.597 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:16:20.696 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:16:21.755 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:16:22.854 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:16:23.957 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:16:25.055 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:16:26.153 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:16:27.251 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:16:28.354 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:16:28.354 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:16:28.396 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:16:29.490 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:18:29.556 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:18:29.596 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:18:30.690 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:18:30.709 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:18:30.709 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:18:30.709 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:18:30.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:18:31.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:18:31.053 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T02:18:46.049 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T02:18:46.050 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:18:46.050 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T02:18:46.051 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T02:18:46.051 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T02:18:46.051 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T02:18:46.051 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T02:18:46.051 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T02:18:46.051 ERROR:paramiko.transport:socket.timeout 2021-03-23T02:18:46.051 ERROR:paramiko.transport: 2021-03-23T02:18:46.051 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T02:18:46.051 ERROR:paramiko.transport: 2021-03-23T02:18:46.051 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:18:46.051 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T02:18:46.052 ERROR:paramiko.transport: self._check_banner() 2021-03-23T02:18:46.052 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T02:18:46.052 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T02:18:46.052 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T02:18:46.052 ERROR:paramiko.transport: 2021-03-23T02:18:46.053 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:18:46.092 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:18:47.182 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:20:47.248 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:20:47.289 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:20:48.422 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:20:48.489 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:20:48.489 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:20:48.489 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:20:48.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:20:48.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:20:48.748 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T02:20:58.161 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:20:58.207 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:20:58.208 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:20:58.321 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:20:58.321 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:20:58.409 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:20:58.438 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T02:20:58.512 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T02:20:58.513 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:20:58.588 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:20:58.619 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T02:20:58.664 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T02:21:00.422 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log: file size changed while zipping 2021-03-23T02:21:02.716 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log: file size changed while zipping 2021-03-23T02:21:07.084 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log: file size changed while zipping 2021-03-23T02:21:07.084 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T02:21:09.135 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:21:09.179 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:21:10.337 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:23:10.403 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:23:19.235 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:23:21.595 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:23:21.613 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:23:21.613 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:23:21.613 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:23:21.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:23:21.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:23:21.863 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T02:23:22.463 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:23:22.498 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:23:22.498 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:23:22.543 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:23:22.543 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:23:22.546 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:23:22.711 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:23:22.739 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T02:23:22.777 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T02:23:22.777 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T02:23:22.777 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T02:23:22.777 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:23:22.819 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:23:24.761 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:25:24.832 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:25:25.694 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:25:27.833 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:25:27.854 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:25:27.855 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:25:27.855 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:25:27.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:25:28.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:25:28.118 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T02:25:43.124 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T02:25:43.125 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:25:43.125 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T02:25:43.125 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T02:25:43.125 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T02:25:43.125 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T02:25:43.125 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T02:25:43.125 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T02:25:43.126 ERROR:paramiko.transport:socket.timeout 2021-03-23T02:25:43.126 ERROR:paramiko.transport: 2021-03-23T02:25:43.126 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T02:25:43.126 ERROR:paramiko.transport: 2021-03-23T02:25:43.126 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:25:43.126 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T02:25:43.126 ERROR:paramiko.transport: self._check_banner() 2021-03-23T02:25:43.126 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T02:25:43.126 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T02:25:43.126 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T02:25:43.126 ERROR:paramiko.transport: 2021-03-23T02:25:43.128 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:25:43.173 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:25:44.302 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:27:44.425 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:27:44.470 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:27:45.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:27:45.571 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:27:45.572 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:27:45.573 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:27:45.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:27:45.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:27:45.861 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T02:28:00.891 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T02:28:00.895 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:28:00.895 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T02:28:00.895 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T02:28:00.895 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T02:28:00.895 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T02:28:00.895 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T02:28:00.895 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T02:28:00.896 ERROR:paramiko.transport:socket.timeout 2021-03-23T02:28:00.896 ERROR:paramiko.transport: 2021-03-23T02:28:00.896 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T02:28:00.896 ERROR:paramiko.transport: 2021-03-23T02:28:00.896 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:28:00.896 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T02:28:00.896 ERROR:paramiko.transport: self._check_banner() 2021-03-23T02:28:00.896 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T02:28:00.896 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T02:28:00.896 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T02:28:00.896 ERROR:paramiko.transport: 2021-03-23T02:28:00.897 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:28:00.945 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:28:03.405 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:30:03.496 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:30:03.565 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:30:05.194 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:30:05.212 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:30:05.212 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:30:05.212 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:30:05.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:30:05.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:30:20.544 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T02:30:20.544 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:30:20.545 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T02:30:20.545 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T02:30:20.545 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T02:30:20.545 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T02:30:20.545 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T02:30:20.545 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T02:30:20.545 ERROR:paramiko.transport:socket.timeout 2021-03-23T02:30:20.545 ERROR:paramiko.transport: 2021-03-23T02:30:20.545 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T02:30:20.546 ERROR:paramiko.transport: 2021-03-23T02:30:20.546 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:30:20.546 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T02:30:20.546 ERROR:paramiko.transport: self._check_banner() 2021-03-23T02:30:20.546 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T02:30:20.546 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T02:30:20.546 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T02:30:20.546 ERROR:paramiko.transport: 2021-03-23T02:30:20.547 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:30:20.591 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:30:21.839 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:32:21.910 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:32:21.964 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:32:24.946 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:32:24.977 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:32:24.977 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:32:24.977 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:32:24.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:32:25.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:32:27.952 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:32:28.073 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:32:28.073 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:32:28.104 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:32:28.132 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T02:32:28.133 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:32:28.193 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:32:30.424 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:34:30.499 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:34:30.552 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:34:32.993 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:34:33.027 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:34:33.027 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:34:33.027 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:34:33.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:34:33.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:34:33.663 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:34:33.721 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:34:33.721 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:34:33.723 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:34:33.759 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T02:34:33.765 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:34:33.811 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:34:35.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:36:35.537 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:36:35.579 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:36:36.783 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:36:36.801 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:36:36.801 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:36:36.802 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:36:36.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:36:37.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:36:52.110 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T02:36:52.111 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:36:52.111 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T02:36:52.112 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T02:36:52.112 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T02:36:52.112 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T02:36:52.112 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T02:36:52.113 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T02:36:52.113 ERROR:paramiko.transport:socket.timeout 2021-03-23T02:36:52.114 ERROR:paramiko.transport: 2021-03-23T02:36:52.114 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T02:36:52.114 ERROR:paramiko.transport: 2021-03-23T02:36:52.114 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:36:52.114 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T02:36:52.114 ERROR:paramiko.transport: self._check_banner() 2021-03-23T02:36:52.114 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T02:36:52.114 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T02:36:52.115 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T02:36:52.115 ERROR:paramiko.transport: 2021-03-23T02:36:52.116 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:36:52.168 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:36:53.307 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:38:53.377 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:38:53.422 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:38:54.567 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:38:54.591 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:38:54.591 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:38:54.591 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:38:54.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:38:54.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:39:08.231 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:39:08.510 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:39:08.510 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:39:08.657 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:39:08.693 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T02:39:08.694 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:39:08.737 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:39:09.861 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:41:09.930 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:41:09.972 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:41:11.116 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:41:11.134 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:41:11.134 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:41:11.135 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:41:11.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:41:11.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:41:11.965 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:41:12.026 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:41:12.027 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:41:12.028 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:41:12.068 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T02:41:12.076 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:41:12.118 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:41:13.245 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:43:13.311 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:43:13.356 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:43:14.543 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:43:14.564 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:43:14.565 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:43:14.565 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:43:14.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:43:14.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:43:29.894 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T02:43:29.895 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:43:29.895 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T02:43:29.895 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T02:43:29.896 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T02:43:29.896 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T02:43:29.896 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T02:43:29.896 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T02:43:29.896 ERROR:paramiko.transport:socket.timeout 2021-03-23T02:43:29.896 ERROR:paramiko.transport: 2021-03-23T02:43:29.896 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T02:43:29.896 ERROR:paramiko.transport: 2021-03-23T02:43:29.897 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:43:29.897 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T02:43:29.897 ERROR:paramiko.transport: self._check_banner() 2021-03-23T02:43:29.897 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T02:43:29.897 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T02:43:29.898 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T02:43:29.898 ERROR:paramiko.transport: 2021-03-23T02:43:29.899 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:43:29.946 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:43:31.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:45:31.194 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:45:31.237 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:45:32.364 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:45:32.382 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:45:32.382 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:45:32.382 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:45:32.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:45:32.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:45:47.695 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T02:45:47.696 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:45:47.696 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T02:45:47.696 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T02:45:47.696 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T02:45:47.696 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T02:45:47.696 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T02:45:47.697 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T02:45:47.697 ERROR:paramiko.transport:socket.timeout 2021-03-23T02:45:47.697 ERROR:paramiko.transport: 2021-03-23T02:45:47.697 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T02:45:47.697 ERROR:paramiko.transport: 2021-03-23T02:45:47.697 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:45:47.697 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T02:45:47.697 ERROR:paramiko.transport: self._check_banner() 2021-03-23T02:45:47.697 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T02:45:47.697 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T02:45:47.697 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T02:45:47.697 ERROR:paramiko.transport: 2021-03-23T02:45:47.698 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:45:47.740 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:45:48.794 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:47:48.863 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:47:48.904 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:47:50.062 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:47:50.080 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:47:50.080 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:47:50.081 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:47:50.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:47:50.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:47:52.360 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:47:52.428 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:47:52.429 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:47:52.439 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:47:52.467 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T02:47:52.485 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:47:52.530 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:47:53.621 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:49:53.697 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:49:53.745 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:49:54.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:49:54.933 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:49:54.934 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:49:54.935 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:49:54.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:49:55.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:49:55.486 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T02:49:55.550 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T02:49:55.550 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T02:49:55.551 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:49:55.586 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T02:49:55.598 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T02:49:55.639 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:49:56.715 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:51:56.782 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:51:56.827 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:51:57.956 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:51:57.974 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:51:57.974 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:51:57.974 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:51:57.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:51:58.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:52:13.271 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T02:52:13.273 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:52:13.273 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T02:52:13.273 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T02:52:13.273 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T02:52:13.273 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T02:52:13.273 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T02:52:13.273 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T02:52:13.273 ERROR:paramiko.transport:socket.timeout 2021-03-23T02:52:13.273 ERROR:paramiko.transport: 2021-03-23T02:52:13.274 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T02:52:13.274 ERROR:paramiko.transport: 2021-03-23T02:52:13.274 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T02:52:13.274 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T02:52:13.274 ERROR:paramiko.transport: self._check_banner() 2021-03-23T02:52:13.274 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T02:52:13.274 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T02:52:13.274 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T02:52:13.274 ERROR:paramiko.transport: 2021-03-23T02:52:13.275 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:52:13.318 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:52:14.510 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:54:14.579 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:54:14.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:54:15.760 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:54:15.777 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:54:15.778 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:54:15.778 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:54:15.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:54:18.850 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:54:18.904 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:54:20.195 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:56:20.278 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:56:20.333 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:56:21.668 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:56:21.686 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:56:21.686 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:56:21.686 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:56:21.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:56:21.826 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:56:22.930 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:56:24.029 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:56:25.108 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:56:26.211 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:56:27.310 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:56:28.373 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:56:29.434 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:56:30.532 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:56:31.596 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T02:56:31.596 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:56:31.635 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:56:32.753 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:58:32.837 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T02:58:32.880 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:58:34.082 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T02:58:34.102 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T02:58:34.102 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T02:58:34.102 INFO:teuthology.misc:Compressing logs... 2021-03-23T02:58:34.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T02:58:41.224 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T02:58:41.266 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T02:58:42.426 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:00:42.494 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:00:42.539 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:00:43.826 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:00:43.844 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:00:43.845 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:00:43.845 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:00:43.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:00:44.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:00:44.225 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:00:59.292 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T03:00:59.293 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:00:59.293 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T03:00:59.293 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T03:00:59.293 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T03:00:59.293 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T03:00:59.293 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T03:00:59.294 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T03:00:59.294 ERROR:paramiko.transport:socket.timeout 2021-03-23T03:00:59.294 ERROR:paramiko.transport: 2021-03-23T03:00:59.294 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T03:00:59.294 ERROR:paramiko.transport: 2021-03-23T03:00:59.294 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:00:59.294 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T03:00:59.294 ERROR:paramiko.transport: self._check_banner() 2021-03-23T03:00:59.294 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T03:00:59.294 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T03:00:59.295 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T03:00:59.295 ERROR:paramiko.transport: 2021-03-23T03:00:59.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:00:59.438 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:01:00.684 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:03:00.778 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:03:00.821 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:03:01.964 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:03:01.987 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:03:01.987 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:03:01.987 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:03:01.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:03:02.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:03:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:03:17.407 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T03:03:17.408 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:03:17.408 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T03:03:17.408 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T03:03:17.408 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T03:03:17.408 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T03:03:17.408 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T03:03:17.408 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T03:03:17.408 ERROR:paramiko.transport:socket.timeout 2021-03-23T03:03:17.409 ERROR:paramiko.transport: 2021-03-23T03:03:17.409 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T03:03:17.409 ERROR:paramiko.transport: 2021-03-23T03:03:17.409 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:03:17.409 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T03:03:17.409 ERROR:paramiko.transport: self._check_banner() 2021-03-23T03:03:17.409 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T03:03:17.409 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T03:03:17.409 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T03:03:17.409 ERROR:paramiko.transport: 2021-03-23T03:03:17.410 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:03:17.451 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:03:18.519 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:05:18.586 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:05:18.637 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:05:19.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:05:19.888 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:05:19.889 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:05:19.889 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:05:19.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:05:38.243 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:05:38.290 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:05:39.494 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:07:39.574 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:07:39.623 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:07:40.842 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:07:40.859 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:07:40.859 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:07:40.859 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:07:40.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:07:41.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:07:41.203 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:07:42.163 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:07:42.188 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:07:42.189 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:07:42.232 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:07:42.233 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:07:42.236 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:07:42.257 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-03-23T03:07:42.407 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:07:42.439 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-23T03:07:42.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T03:07:42.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T03:07:42.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T03:07:42.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T03:07:42.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T03:07:42.453 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T03:07:42.499 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:07:43.614 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:09:43.679 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:09:43.719 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:09:44.766 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:09:44.782 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:09:44.783 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:09:44.783 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:09:44.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:09:45.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:09:45.146 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:10:26.753 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:10:27.042 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:10:27.042 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:10:27.241 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:10:29.696 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:10:30.416 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T03:10:30.416 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T03:10:30.417 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T03:10:30.417 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T03:10:30.417 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T03:10:30.417 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T03:10:30.458 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:10:31.594 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:12:31.662 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:12:31.701 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:12:32.740 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:12:32.762 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:12:32.762 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:12:32.762 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:12:32.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:12:33.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:12:48.164 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T03:12:48.164 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:12:48.164 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T03:12:48.165 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T03:12:48.165 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T03:12:48.165 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T03:12:48.165 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T03:12:48.165 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T03:12:48.166 ERROR:paramiko.transport:socket.timeout 2021-03-23T03:12:48.166 ERROR:paramiko.transport: 2021-03-23T03:12:48.166 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T03:12:48.166 ERROR:paramiko.transport: 2021-03-23T03:12:48.166 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:12:48.166 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T03:12:48.166 ERROR:paramiko.transport: self._check_banner() 2021-03-23T03:12:48.166 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T03:12:48.166 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T03:12:48.167 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T03:12:48.168 ERROR:paramiko.transport: 2021-03-23T03:12:48.168 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:12:48.209 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:12:49.292 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:14:49.366 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:14:49.409 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:14:50.516 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:14:50.535 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:14:50.536 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:14:50.536 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:14:50.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:14:50.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:15:05.973 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T03:15:05.974 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:15:05.974 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T03:15:05.974 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T03:15:05.974 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T03:15:05.975 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T03:15:05.975 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T03:15:05.975 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T03:15:05.975 ERROR:paramiko.transport:socket.timeout 2021-03-23T03:15:05.975 ERROR:paramiko.transport: 2021-03-23T03:15:05.975 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T03:15:05.975 ERROR:paramiko.transport: 2021-03-23T03:15:05.975 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:15:05.975 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T03:15:05.975 ERROR:paramiko.transport: self._check_banner() 2021-03-23T03:15:05.975 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T03:15:05.975 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T03:15:05.976 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T03:15:05.976 ERROR:paramiko.transport: 2021-03-23T03:15:05.977 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:15:06.023 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:15:07.452 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:17:07.544 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:17:07.588 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:17:08.908 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:17:08.928 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:17:08.928 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:17:08.929 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:17:08.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:17:09.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:17:24.388 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T03:17:24.390 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:17:24.391 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T03:17:24.391 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T03:17:24.391 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T03:17:24.391 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T03:17:24.392 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T03:17:24.392 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T03:17:24.392 ERROR:paramiko.transport:socket.timeout 2021-03-23T03:17:24.392 ERROR:paramiko.transport: 2021-03-23T03:17:24.393 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T03:17:24.393 ERROR:paramiko.transport: 2021-03-23T03:17:24.393 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:17:24.393 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T03:17:24.394 ERROR:paramiko.transport: self._check_banner() 2021-03-23T03:17:24.394 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T03:17:24.394 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T03:17:24.394 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T03:17:24.395 ERROR:paramiko.transport: 2021-03-23T03:17:24.395 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:17:24.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:17:26.962 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:19:27.040 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:19:32.702 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:19:33.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:19:33.960 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:19:33.961 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:19:33.961 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:19:33.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:19:34.041 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T03:19:35.142 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T03:19:36.250 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T03:19:37.359 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T03:19:38.464 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T03:19:39.560 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T03:19:40.807 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T03:19:41.904 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T03:19:43.023 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T03:19:44.131 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T03:19:44.131 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:19:44.172 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:19:45.273 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:21:45.355 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:21:45.407 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:21:46.572 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:21:46.590 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:21:46.590 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:21:46.591 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:21:46.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:21:46.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:21:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T03:25:37.871 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:25:37.958 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:25:37.959 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:25:38.657 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:25:38.658 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:25:38.954 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:26:14.335 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:26:14.366 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T03:26:46.594 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T03:26:46.594 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T03:26:46.595 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T03:26:46.595 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T03:26:46.595 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T03:26:46.595 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T03:26:46.642 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:26:47.740 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:28:47.814 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:28:47.876 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:28:49.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:28:49.088 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:28:49.089 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:28:49.089 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:28:49.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:28:49.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:28:52.731 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:28:52.871 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:28:52.871 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:28:52.908 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:28:53.168 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:28:53.319 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T03:28:53.319 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T03:28:53.319 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T03:28:53.319 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T03:28:53.319 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T03:28:53.320 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T03:28:53.370 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:28:54.472 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:30:54.538 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:30:54.583 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:30:55.694 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:30:55.714 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:30:55.714 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:30:55.715 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:30:55.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:30:55.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:30:56.322 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:30:56.381 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:30:56.381 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:30:56.383 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:30:56.551 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:30:56.589 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T03:30:56.589 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T03:30:56.589 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T03:30:56.589 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T03:30:56.589 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T03:30:56.598 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T03:30:56.640 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:30:57.708 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:32:57.775 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:32:57.817 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:32:58.939 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:32:58.957 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:32:58.958 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:32:58.958 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:32:58.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:32:59.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:33:14.220 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T03:33:14.221 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:33:14.221 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T03:33:14.221 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T03:33:14.221 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T03:33:14.221 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T03:33:14.222 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T03:33:14.222 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T03:33:14.222 ERROR:paramiko.transport:socket.timeout 2021-03-23T03:33:14.222 ERROR:paramiko.transport: 2021-03-23T03:33:14.222 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T03:33:14.222 ERROR:paramiko.transport: 2021-03-23T03:33:14.222 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:33:14.222 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T03:33:14.222 ERROR:paramiko.transport: self._check_banner() 2021-03-23T03:33:14.222 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T03:33:14.223 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T03:33:14.223 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T03:33:14.223 ERROR:paramiko.transport: 2021-03-23T03:33:14.225 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:33:14.280 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:33:15.446 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:35:15.538 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:35:15.579 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:35:16.690 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:35:16.707 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:35:16.707 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:35:16.708 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:35:16.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:35:16.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:35:32.003 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T03:35:32.004 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:35:32.005 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T03:35:32.005 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T03:35:32.005 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T03:35:32.005 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T03:35:32.005 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T03:35:32.005 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T03:35:32.005 ERROR:paramiko.transport:socket.timeout 2021-03-23T03:35:32.005 ERROR:paramiko.transport: 2021-03-23T03:35:32.005 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T03:35:32.006 ERROR:paramiko.transport: 2021-03-23T03:35:32.006 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:35:32.006 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T03:35:32.006 ERROR:paramiko.transport: self._check_banner() 2021-03-23T03:35:32.006 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T03:35:32.006 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T03:35:32.006 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T03:35:32.006 ERROR:paramiko.transport: 2021-03-23T03:35:32.007 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:35:32.052 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:35:33.148 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:37:33.213 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:37:33.255 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:37:34.380 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:37:34.398 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:37:34.398 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:37:34.398 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:37:34.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:37:34.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:37:49.669 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T03:37:49.670 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:37:49.671 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T03:37:49.671 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T03:37:49.671 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T03:37:49.671 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T03:37:49.671 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T03:37:49.672 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T03:37:49.672 ERROR:paramiko.transport:socket.timeout 2021-03-23T03:37:49.672 ERROR:paramiko.transport: 2021-03-23T03:37:49.672 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T03:37:49.672 ERROR:paramiko.transport: 2021-03-23T03:37:49.672 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:37:49.672 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T03:37:49.672 ERROR:paramiko.transport: self._check_banner() 2021-03-23T03:37:49.672 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T03:37:49.672 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T03:37:49.672 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T03:37:49.673 ERROR:paramiko.transport: 2021-03-23T03:37:49.673 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:37:49.715 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:37:50.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:39:50.882 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:39:50.922 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:39:52.007 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:39:52.024 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:39:52.025 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:39:52.025 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:39:52.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:39:52.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:40:07.793 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:40:08.045 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:40:08.045 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:40:08.322 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:40:08.594 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:40:08.763 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T03:40:08.763 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T03:40:08.763 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T03:40:08.763 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T03:40:08.763 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T03:40:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2021-03-23T03:40:11.772 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping 2021-03-23T03:40:13.634 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping 2021-03-23T03:40:14.340 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2021-03-23T03:40:14.342 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T03:40:14.395 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:40:15.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:42:15.545 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:42:15.584 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:42:16.678 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:42:16.696 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:42:16.696 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:42:16.696 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:42:16.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:42:51.330 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:42:51.370 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:42:52.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:44:52.600 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:44:52.641 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:44:53.732 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:44:53.823 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:44:53.823 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:44:53.823 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:44:53.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:44:56.899 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:44:56.938 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:44:58.001 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:46:58.068 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:46:58.117 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:46:59.227 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:46:59.249 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:46:59.249 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:46:59.249 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:46:59.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:47:02.310 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:47:02.355 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:47:03.466 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:49:03.533 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:49:07.992 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:49:10.573 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:49:10.592 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:49:10.592 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:49:10.593 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:49:10.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:49:11.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:49:11.129 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:49:47.732 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:49:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:49:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:49:48.088 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:49:48.088 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:49:48.347 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:49:48.364 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-23T03:49:48.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T03:49:48.602 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:49:48.895 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:49:48.920 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-23T03:49:49.813 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T03:49:49.813 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T03:49:49.813 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T03:49:49.813 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T03:49:49.813 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T03:49:49.814 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T03:49:49.886 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:49:51.238 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:51:51.307 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:52:06.449 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:52:08.270 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:52:08.292 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:52:08.293 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:52:08.293 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:52:08.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:52:23.747 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:52:23.811 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:52:25.391 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:54:25.483 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:54:25.681 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:54:28.316 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:54:28.346 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:54:28.346 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:54:28.346 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:54:28.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:54:28.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:54:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:54:43.706 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T03:54:43.707 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:54:43.707 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T03:54:43.707 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T03:54:43.707 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T03:54:43.707 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T03:54:43.707 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T03:54:43.707 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T03:54:43.708 ERROR:paramiko.transport:socket.timeout 2021-03-23T03:54:43.708 ERROR:paramiko.transport: 2021-03-23T03:54:43.708 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T03:54:43.708 ERROR:paramiko.transport: 2021-03-23T03:54:43.708 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T03:54:43.708 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T03:54:43.708 ERROR:paramiko.transport: self._check_banner() 2021-03-23T03:54:43.708 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T03:54:43.708 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T03:54:43.708 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T03:54:43.709 ERROR:paramiko.transport: 2021-03-23T03:54:43.709 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T03:54:43.763 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:54:45.422 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:56:45.486 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:56:45.767 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:56:47.502 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:56:47.521 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:56:47.521 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:56:47.521 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:56:47.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:56:47.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:56:47.884 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:56:49.096 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:56:49.124 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:56:49.125 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:56:49.163 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:56:49.164 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:56:49.168 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:56:49.369 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:56:49.388 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-23T03:56:49.394 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T03:56:49.394 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T03:56:49.394 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T03:56:49.395 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T03:56:49.395 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T03:56:49.395 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T03:56:49.440 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:56:50.611 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:58:50.681 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T03:58:55.004 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:58:56.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T03:58:56.084 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T03:58:56.085 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T03:58:56.085 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:58:56.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:58:56.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T03:58:56.747 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T03:58:56.806 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T03:58:56.807 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T03:58:56.810 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:58:56.958 INFO:teuthology.misc:Compressing logs... 2021-03-23T03:58:56.999 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T03:58:56.999 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T03:58:57.000 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T03:58:57.000 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T03:58:57.000 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T03:58:57.022 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T03:58:57.063 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T03:58:58.244 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:00:58.310 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:00:58.367 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:01:00.597 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:01:00.621 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:01:00.622 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:01:00.622 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:01:00.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:01:01.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:01:16.039 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T04:01:16.040 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T04:01:16.040 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T04:01:16.040 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T04:01:16.040 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T04:01:16.040 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T04:01:16.040 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T04:01:16.041 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T04:01:16.041 ERROR:paramiko.transport:socket.timeout 2021-03-23T04:01:16.041 ERROR:paramiko.transport: 2021-03-23T04:01:16.041 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T04:01:16.041 ERROR:paramiko.transport: 2021-03-23T04:01:16.041 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T04:01:16.041 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T04:01:16.041 ERROR:paramiko.transport: self._check_banner() 2021-03-23T04:01:16.041 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T04:01:16.041 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T04:01:16.041 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T04:01:16.042 ERROR:paramiko.transport: 2021-03-23T04:01:16.042 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T04:01:16.099 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:01:17.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:03:17.778 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:03:19.742 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:03:21.865 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:03:21.882 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:03:21.882 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:03:21.882 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:03:21.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:03:22.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:03:23.161 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:03:23.212 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:03:23.212 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:03:23.215 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:03:23.413 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:03:23.432 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists; not overwritten 2021-03-23T04:03:23.475 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:03:23.475 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:03:23.475 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:03:23.476 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:03:23.476 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:03:23.605 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2021-03-23T04:03:23.606 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-23T04:03:24.454 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2021-03-23T04:03:24.454 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-23T04:03:24.455 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:03:24.499 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:03:26.524 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:05:26.592 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:05:26.641 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:05:27.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:05:27.987 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:05:27.987 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:05:27.988 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:05:27.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:05:28.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:05:43.428 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T04:05:43.428 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T04:05:43.429 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T04:05:43.429 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T04:05:43.429 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T04:05:43.429 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T04:05:43.429 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T04:05:43.430 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T04:05:43.430 ERROR:paramiko.transport:socket.timeout 2021-03-23T04:05:43.430 ERROR:paramiko.transport: 2021-03-23T04:05:43.430 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T04:05:43.430 ERROR:paramiko.transport: 2021-03-23T04:05:43.431 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T04:05:43.431 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T04:05:43.431 ERROR:paramiko.transport: self._check_banner() 2021-03-23T04:05:43.431 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T04:05:43.431 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T04:05:43.431 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T04:05:43.432 ERROR:paramiko.transport: 2021-03-23T04:05:43.433 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T04:05:43.478 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:05:44.949 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:07:45.022 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:07:45.083 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:07:46.461 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:07:46.478 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:07:46.479 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:07:46.479 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:07:46.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:07:46.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:08:01.924 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T04:08:01.926 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T04:08:01.926 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T04:08:01.926 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T04:08:01.926 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T04:08:01.927 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T04:08:01.927 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T04:08:01.927 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T04:08:01.927 ERROR:paramiko.transport:socket.timeout 2021-03-23T04:08:01.927 ERROR:paramiko.transport: 2021-03-23T04:08:01.927 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T04:08:01.927 ERROR:paramiko.transport: 2021-03-23T04:08:01.927 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T04:08:01.927 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T04:08:01.927 ERROR:paramiko.transport: self._check_banner() 2021-03-23T04:08:01.928 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T04:08:01.928 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T04:08:01.928 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T04:08:01.928 ERROR:paramiko.transport: 2021-03-23T04:08:01.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T04:08:01.971 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:08:03.111 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:10:03.180 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:10:03.228 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:10:04.482 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:10:04.499 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:10:04.499 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:10:04.499 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:10:04.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:10:04.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:10:06.486 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:10:06.554 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:10:06.554 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:10:06.557 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:10:06.790 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:10:06.812 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists; not overwritten 2021-03-23T04:10:06.813 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:10:06.813 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:10:06.813 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-23T04:10:06.813 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.7.log.gz already exists; not overwritten 2021-03-23T04:10:06.813 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-23T04:10:06.815 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:10:06.855 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:10:07.982 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:12:08.049 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:12:08.096 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:12:10.401 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:12:10.419 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:12:10.419 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:12:10.419 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:12:10.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:12:10.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:14:06.992 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:14:07.208 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:14:07.209 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:14:07.344 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:14:07.651 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:14:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists; not overwritten 2021-03-23T04:14:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:14:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:14:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-23T04:14:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.7.log.gz already exists; not overwritten 2021-03-23T04:14:07.676 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-23T04:14:07.676 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:14:07.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:14:08.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:16:08.922 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:16:08.969 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:16:11.672 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:16:11.691 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:16:11.692 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:16:11.692 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:16:11.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:16:12.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:16:12.694 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:16:12.756 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:16:12.758 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:16:12.759 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:16:12.901 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:16:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists; not overwritten 2021-03-23T04:16:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:16:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:16:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-23T04:16:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.7.log.gz already exists; not overwritten 2021-03-23T04:16:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-23T04:16:12.938 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:16:12.980 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:16:14.887 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:18:14.980 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:18:15.030 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:18:17.196 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:18:17.216 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:18:17.216 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:18:17.216 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:18:17.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:18:17.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:18:32.661 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-03-23T04:18:32.662 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T04:18:32.663 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-03-23T04:18:32.663 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-03-23T04:18:32.663 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-03-23T04:18:32.663 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-03-23T04:18:32.664 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2021-03-23T04:18:32.664 ERROR:paramiko.transport: raise socket.timeout() 2021-03-23T04:18:32.664 ERROR:paramiko.transport:socket.timeout 2021-03-23T04:18:32.664 ERROR:paramiko.transport: 2021-03-23T04:18:32.664 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-03-23T04:18:32.664 ERROR:paramiko.transport: 2021-03-23T04:18:32.664 ERROR:paramiko.transport:Traceback (most recent call last): 2021-03-23T04:18:32.664 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-03-23T04:18:32.665 ERROR:paramiko.transport: self._check_banner() 2021-03-23T04:18:32.665 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-03-23T04:18:32.665 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-03-23T04:18:32.665 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-03-23T04:18:32.665 ERROR:paramiko.transport: 2021-03-23T04:18:32.666 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T04:18:32.707 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:18:33.884 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:20:33.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:20:34.038 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:20:35.375 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:20:35.393 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:20:35.394 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:20:35.394 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:20:35.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:20:35.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:20:36.695 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:20:36.780 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:20:36.780 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:20:36.784 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:20:36.966 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:20:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists; not overwritten 2021-03-23T04:20:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:20:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:20:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2021-03-23T04:20:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.7.log.gz already exists; not overwritten 2021-03-23T04:20:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2021-03-23T04:20:36.994 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:20:37.040 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:20:38.376 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:22:38.477 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:22:38.568 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:22:39.927 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:22:39.947 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:22:39.947 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:22:39.947 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:22:39.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:22:43.042 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T04:22:43.133 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:22:44.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:24:44.786 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:24:44.838 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:24:47.727 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:24:47.761 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:24:47.763 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:24:47.763 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:24:47.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:24:47.857 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T04:24:48.926 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T04:24:50.040 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T04:24:51.249 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T04:24:52.312 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T04:24:53.415 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T04:24:54.535 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T04:24:55.647 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T04:24:56.762 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T04:24:57.867 ERROR:teuthology.orchestra.connection:Error authenticating with smithi060.front.sepia.ceph.com: Authentication failed. 2021-03-23T04:24:57.868 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T04:24:57.912 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:24:59.309 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:26:59.395 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:26:59.452 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:27:01.927 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:27:01.972 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:27:01.972 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:27:01.972 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:27:01.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:27:05.026 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T04:27:05.081 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:27:07.649 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:29:07.722 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:29:07.767 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:29:09.765 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:29:09.785 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:29:09.786 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:29:09.786 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:29:09.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:29:10.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:29:10.059 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:29:10.375 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:29:10.408 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:29:10.408 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:29:10.438 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:29:10.438 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:29:10.439 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:29:10.458 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2021-03-23T04:29:10.526 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T04:29:10.526 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:29:10.580 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:29:10.596 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-23T04:29:10.637 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:29:10.638 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:29:10.638 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:29:10.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:29:10.639 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:29:10.641 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:29:10.685 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:29:11.921 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:31:12.035 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:31:12.100 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:31:13.635 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:31:13.654 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:31:13.655 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:31:13.655 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:31:13.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:31:13.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:31:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:31:14.276 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:31:14.347 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:31:14.347 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:31:14.369 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:31:14.369 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:31:14.371 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:31:14.546 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:31:14.587 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-23T04:31:14.602 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:31:14.603 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:31:14.603 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:31:14.603 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:31:14.603 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:31:14.606 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:31:14.650 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:31:16.659 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:33:16.726 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:33:16.770 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:33:18.805 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:33:18.822 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:33:18.823 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:33:18.823 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:33:18.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:33:19.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:33:19.124 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:33:19.478 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:33:19.516 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:33:19.517 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:33:19.538 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:33:19.538 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:33:19.542 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:33:19.756 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:33:19.812 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-23T04:33:19.827 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:33:19.827 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:33:19.827 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:33:19.827 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:33:19.827 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:33:19.828 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:33:19.883 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:33:21.084 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:35:21.154 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:35:21.200 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:35:22.402 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:35:22.419 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:35:22.420 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:35:22.420 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:35:22.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:35:22.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:35:22.786 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:35:23.239 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:35:23.281 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:35:23.282 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:35:23.313 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:35:23.313 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:35:23.315 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:35:23.600 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:35:23.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:35:23.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:35:23.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:35:23.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:35:23.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:35:23.678 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-23T04:35:23.682 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:35:23.735 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:35:25.037 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:37:25.107 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:37:25.151 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:37:27.240 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:37:27.266 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:37:27.267 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:37:27.267 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:37:27.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:37:27.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:37:27.578 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:37:27.874 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:37:27.916 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:37:27.916 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:37:27.935 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:37:27.935 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:37:27.937 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:37:28.148 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:37:28.183 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:37:28.184 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:37:28.184 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:37:28.184 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:37:28.184 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:37:28.188 INFO:teuthology.orchestra.run.smithi060.stderr:find: '/var/log/ceph': No such file or directory 2021-03-23T04:37:28.199 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:37:28.243 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:37:29.417 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:39:29.485 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:39:32.521 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:39:35.177 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:39:35.226 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:39:35.226 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:39:35.226 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:39:35.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:39:35.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:39:36.007 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:39:36.210 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:39:36.213 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:39:36.215 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:39:36.505 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:39:36.538 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:39:36.538 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:39:36.538 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:39:36.538 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:39:36.539 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:39:36.580 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:39:36.657 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:39:39.675 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:41:39.781 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:41:39.822 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:41:41.711 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:41:41.728 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:41:41.728 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:41:41.729 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:41:41.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:41:42.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:41:42.495 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:41:42.598 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:41:42.599 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:41:42.601 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:41:42.952 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:41:43.012 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:41:43.012 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:41:43.012 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:41:43.012 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:41:43.013 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:41:43.192 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:41:43.241 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:41:44.436 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:43:44.544 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:43:44.599 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:43:46.866 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:43:46.885 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:43:46.885 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:43:46.885 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:43:46.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:43:47.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:43:47.508 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:43:47.579 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:43:47.580 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:43:47.582 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:43:47.761 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:43:47.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:43:47.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:43:47.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:43:47.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:43:47.796 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:43:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T04:43:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T04:43:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:43:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:43:47.814 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:43:47.859 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:43:49.850 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:45:49.917 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:45:50.413 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:45:52.511 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:45:52.530 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:45:52.530 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:45:52.531 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:45:52.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:45:52.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:45:53.149 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:45:53.224 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:45:53.224 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:45:53.232 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:45:53.394 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:45:53.443 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:45:53.444 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:45:53.444 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:45:53.444 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:45:53.444 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:45:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T04:45:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T04:45:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:45:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:45:53.451 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:45:53.499 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:45:54.714 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:47:54.780 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:47:54.828 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:47:57.502 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:47:57.526 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:47:57.527 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:47:57.527 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:47:57.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:47:57.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:47:58.142 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:47:58.215 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:47:58.215 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:47:58.217 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:47:58.407 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:47:58.449 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:47:58.450 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:47:58.450 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:47:58.450 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:47:58.450 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:47:58.455 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T04:47:58.455 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T04:47:58.455 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:47:58.455 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:47:58.462 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:47:58.527 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:48:01.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:50:01.800 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:50:03.880 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:50:05.984 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:50:06.001 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:50:06.002 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:50:06.002 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:50:06.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:50:06.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:50:06.625 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:50:06.704 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:50:06.704 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:50:06.706 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:50:06.915 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:50:06.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T04:50:06.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T04:50:06.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:50:06.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:50:06.967 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:50:06.967 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:50:06.967 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:50:06.967 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:50:06.967 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:50:06.968 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:50:07.017 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:50:08.198 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:52:08.267 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:52:08.316 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:52:10.276 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:52:10.293 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:52:10.293 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:52:10.293 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:52:10.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:52:10.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:52:10.917 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:52:10.990 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:52:10.990 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:52:10.993 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:52:11.167 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:52:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T04:52:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T04:52:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:52:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:52:11.221 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:52:11.267 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:52:12.419 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:54:12.492 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:54:12.539 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:54:14.489 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:54:14.506 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:54:14.506 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:54:14.507 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:54:14.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:54:14.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:54:15.117 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:54:15.190 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:54:15.190 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:54:15.192 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:54:15.379 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:54:15.426 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:54:15.426 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:54:15.426 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:54:15.427 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:54:15.427 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:54:15.429 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T04:54:15.429 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T04:54:15.429 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:54:15.429 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:54:15.435 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:54:15.476 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:54:16.612 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:56:16.679 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:56:16.729 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:56:20.081 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:56:20.130 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:56:20.130 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:56:20.130 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:56:20.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:56:20.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:56:20.775 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:56:20.847 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:56:20.848 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:56:20.850 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:56:21.040 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:56:21.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T04:56:21.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T04:56:21.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T04:56:21.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T04:56:21.076 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T04:56:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T04:56:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T04:56:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:56:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:56:21.104 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:56:21.161 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:56:22.351 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:58:22.422 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T04:58:22.468 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:58:24.531 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T04:58:24.560 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T04:58:24.561 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T04:58:24.561 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:58:24.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:58:24.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T04:58:25.256 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T04:58:25.327 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T04:58:25.327 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T04:58:25.329 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:58:25.548 INFO:teuthology.misc:Compressing logs... 2021-03-23T04:58:25.594 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T04:58:25.594 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T04:58:25.595 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T04:58:25.595 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T04:58:25.601 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T04:58:25.756 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T04:58:27.145 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:00:27.259 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:00:27.309 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:00:29.944 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:00:29.967 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:00:29.968 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:00:29.968 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:00:29.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:00:30.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:00:30.658 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:00:30.757 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:00:30.757 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:00:30.762 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:00:30.967 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:00:31.016 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:00:31.017 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:00:31.017 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:00:31.018 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:00:31.021 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:00:31.021 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:00:31.021 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:00:31.022 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:00:31.022 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:00:31.034 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:00:31.115 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:00:32.525 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:02:32.622 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:02:32.666 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:02:34.924 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:02:34.949 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:02:34.950 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:02:34.950 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:02:34.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:02:35.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:02:35.616 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:02:35.685 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:02:35.685 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:02:35.691 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:02:35.871 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:02:35.918 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:02:35.918 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:02:35.918 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:02:35.919 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:02:35.919 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:02:35.927 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:02:35.927 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:02:35.928 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:02:35.928 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:02:35.932 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:02:35.975 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:02:37.170 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:04:37.269 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:04:37.311 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:04:39.543 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:04:39.560 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:04:39.561 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:04:39.561 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:04:39.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:04:39.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:04:40.183 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:04:40.365 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:04:40.366 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:04:40.369 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:04:40.607 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:04:40.710 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:04:40.711 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:04:40.713 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:04:40.713 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:04:40.714 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:04:40.720 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:04:40.720 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:04:40.720 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:04:40.720 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:04:40.737 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:04:40.808 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:04:43.357 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:06:43.470 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:06:43.518 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:06:45.547 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:06:45.572 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:06:45.572 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:06:45.572 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:06:45.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:06:45.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:06:46.431 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:06:46.503 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:06:46.503 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:06:46.508 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:06:46.710 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:06:46.762 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:06:46.763 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:06:46.763 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:06:46.763 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:06:46.763 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:06:46.784 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:06:46.784 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:06:46.784 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:06:46.784 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:06:46.790 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:06:46.838 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:06:48.184 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:08:48.295 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:08:48.337 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:08:49.561 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:08:49.590 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:08:49.591 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:08:49.591 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:08:49.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:08:49.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:08:50.207 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:08:50.287 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:08:50.288 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:08:50.291 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:08:50.459 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:08:50.501 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:08:50.501 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:08:50.501 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:08:50.502 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:08:50.502 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:08:50.513 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:08:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:08:50.515 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:08:50.515 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:08:50.517 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:08:50.557 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:08:51.726 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:10:51.810 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:10:51.856 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:10:53.000 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:10:53.019 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:10:53.019 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:10:53.019 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:10:53.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:10:53.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:10:53.635 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:10:53.705 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:10:53.705 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:10:53.707 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:10:53.897 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:10:53.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:10:53.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:10:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:10:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:10:53.945 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:10:53.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:10:55.122 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:12:55.195 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:12:55.240 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:12:56.369 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:12:56.390 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:12:56.390 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:12:56.391 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:12:56.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:12:56.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:12:57.015 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:12:57.082 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:12:57.082 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:12:57.084 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:12:57.292 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:12:57.333 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:12:57.334 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:12:57.334 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:12:57.334 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:12:57.338 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:12:57.384 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:12:58.488 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:14:58.557 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:14:58.602 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:14:59.733 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:14:59.758 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:14:59.759 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:14:59.759 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:14:59.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:15:00.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:15:00.355 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:15:00.421 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:15:00.421 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:15:00.423 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:15:00.613 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:15:00.647 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:15:00.648 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:15:00.648 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:15:00.648 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:15:00.648 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:15:00.667 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:15:00.668 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:15:00.668 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:15:00.668 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:15:00.673 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:15:00.717 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:15:01.819 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:17:01.890 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:17:01.941 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:17:03.082 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:17:03.100 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:17:03.100 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:17:03.100 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:17:03.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:17:03.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:17:03.698 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:17:03.764 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:17:03.765 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:17:03.767 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:17:03.932 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:17:03.993 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:17:03.993 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:17:03.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:17:03.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:17:03.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:17:03.995 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:17:03.995 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:17:03.995 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:17:03.995 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:17:04.003 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:17:04.047 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:17:05.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:19:05.336 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:19:05.383 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:19:06.654 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:19:06.671 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:19:06.672 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:19:06.672 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:19:06.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:19:06.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:19:07.302 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:19:07.370 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:19:07.371 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:19:07.373 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:19:07.565 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:19:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:19:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:19:07.610 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:19:07.610 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:19:07.615 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:19:07.676 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:19:08.862 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:21:08.932 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:21:08.980 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:21:10.243 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:21:10.265 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:21:10.266 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:21:10.266 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:21:10.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:21:10.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:21:10.884 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:21:10.953 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:21:10.953 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:21:10.954 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:21:11.158 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:21:11.193 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:21:11.193 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:21:11.193 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:21:11.193 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:21:11.193 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:21:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:21:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:21:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:21:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:21:11.229 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:21:11.286 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:21:12.610 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:23:12.683 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:23:12.725 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:23:14.034 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:23:14.057 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:23:14.058 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:23:14.058 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:23:14.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:23:14.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:23:14.769 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:23:14.861 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:23:14.861 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:23:14.862 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:23:15.264 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:23:15.338 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:23:15.338 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:23:15.339 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:23:15.339 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:23:15.339 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:23:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:23:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:23:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:23:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:23:15.358 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:23:15.414 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:23:16.651 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:25:16.725 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:25:16.765 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:25:17.949 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:25:17.967 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:25:17.968 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:25:17.968 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:25:17.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:25:18.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:25:18.610 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:25:18.684 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:25:18.684 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:25:18.687 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:25:18.965 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:25:19.007 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:25:19.007 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:25:19.007 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:25:19.008 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:25:19.008 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:25:19.027 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:25:19.027 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:25:19.027 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:25:19.027 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:25:19.027 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:25:19.090 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:25:20.417 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:27:20.490 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:27:20.533 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:27:21.649 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:27:21.682 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:27:21.682 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:27:21.682 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:27:21.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:27:21.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:27:22.300 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:27:22.366 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:27:22.366 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:27:22.367 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:27:22.544 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:27:22.585 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:27:22.586 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:27:22.586 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:27:22.586 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:27:22.586 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:27:22.598 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:27:22.598 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:27:22.598 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:27:22.598 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:27:22.605 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:27:22.649 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:27:23.807 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:29:23.879 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:29:23.923 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:29:26.142 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:29:26.159 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:29:26.159 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:29:26.160 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:29:26.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:29:26.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:29:26.763 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:29:26.830 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:29:26.830 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:29:26.832 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:29:27.005 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:29:27.049 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:29:27.049 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:29:27.049 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:29:27.050 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:29:27.050 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:29:27.065 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:29:27.065 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:29:27.066 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:29:27.066 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:29:27.070 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:29:27.119 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:29:29.195 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:31:29.270 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:31:29.316 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:31:31.412 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:31:31.436 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:31:31.436 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:31:31.437 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:31:31.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:31:31.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:31:32.088 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:31:32.157 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:31:32.158 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:31:32.160 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:31:32.374 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:31:32.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:31:32.432 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:31:32.433 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:31:32.433 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:31:32.434 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:31:32.447 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:31:32.448 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:31:32.448 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:31:32.448 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:31:32.452 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:31:32.498 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:31:34.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:33:34.432 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:33:34.476 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:33:37.070 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:33:37.094 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:33:37.094 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:33:37.094 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:33:37.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:33:37.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:33:37.707 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:33:37.776 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:33:37.776 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:33:37.778 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:33:37.954 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:33:37.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:33:37.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:33:37.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:33:37.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:33:37.995 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:33:38.010 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:33:38.011 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:33:38.011 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:33:38.011 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:33:38.015 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:33:38.140 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:33:39.371 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:35:39.445 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:35:39.492 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:35:40.617 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:35:40.634 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:35:40.635 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:35:40.635 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:35:40.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:35:40.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:35:41.239 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:35:41.306 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:35:41.306 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:35:41.309 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:35:41.527 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:35:41.577 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:35:41.577 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:35:41.577 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:35:41.577 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:35:41.585 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:35:41.625 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:35:42.778 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:37:42.849 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:37:42.892 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:37:44.034 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:37:44.050 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:37:44.051 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:37:44.051 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:37:44.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:37:44.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:37:44.711 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:37:44.778 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:37:44.779 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:37:44.781 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:37:44.958 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:37:45.008 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:37:45.008 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:37:45.008 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:37:45.009 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:37:45.011 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:37:45.061 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:37:46.312 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:39:46.378 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:39:46.422 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:39:48.654 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:39:48.672 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:39:48.672 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:39:48.672 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:39:48.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:39:48.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:39:49.320 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:39:49.403 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:39:49.404 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:39:49.405 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:39:49.620 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:39:49.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:39:49.670 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:39:49.670 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:39:49.670 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:39:49.670 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:39:49.689 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:39:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:39:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:39:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:39:49.692 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:39:49.741 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:39:51.749 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:41:51.823 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:41:51.864 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:41:54.005 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:41:54.023 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:41:54.023 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:41:54.023 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:41:54.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:41:54.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:41:54.699 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:41:54.792 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:41:54.792 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:41:54.795 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:41:54.969 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:41:55.016 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:41:55.016 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:41:55.016 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:41:55.017 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:41:55.022 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:41:55.063 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:41:56.838 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:43:56.917 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:43:56.961 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:43:59.072 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:43:59.090 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:43:59.090 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:43:59.090 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:43:59.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:43:59.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:43:59.775 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:43:59.848 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:43:59.849 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:43:59.850 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:44:00.031 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:44:00.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:44:00.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:44:00.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:44:00.076 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:44:00.076 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:44:00.088 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:44:00.089 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:44:00.089 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:44:00.089 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:44:00.094 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:44:00.144 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:44:01.780 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:46:01.849 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:46:01.892 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:46:04.060 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:46:04.082 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:46:04.083 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:46:04.083 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:46:04.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:46:04.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:46:04.775 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:46:04.844 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:46:04.844 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:46:04.846 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:46:05.036 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:46:05.074 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:46:05.074 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:46:05.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:46:05.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:46:05.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:46:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:46:05.093 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:46:05.093 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:46:05.093 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:46:05.098 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:46:05.156 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:46:07.300 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:48:07.370 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:48:07.415 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:48:09.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:48:09.540 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:48:09.540 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:48:09.540 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:48:09.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:48:09.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:48:10.148 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:48:10.220 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:48:10.221 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:48:10.223 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:48:10.387 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:48:10.428 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:48:10.428 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:48:10.428 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:48:10.428 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:48:10.428 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:48:10.451 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:48:10.451 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:48:10.451 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:48:10.451 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:48:10.456 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:48:10.497 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:48:11.647 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:50:11.746 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:50:11.790 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:50:13.777 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:50:13.796 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:50:13.796 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:50:13.796 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:50:13.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:50:14.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:50:14.467 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:50:14.536 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:50:14.536 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:50:14.539 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:50:14.712 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:50:14.768 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:50:14.769 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:50:14.769 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:50:14.769 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:50:14.769 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:50:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:50:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:50:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:50:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:50:14.776 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:50:14.827 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:50:16.671 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:52:16.744 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:52:16.788 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:52:18.711 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:52:18.728 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:52:18.728 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:52:18.729 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:52:18.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:52:19.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:52:19.435 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:52:19.507 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:52:19.507 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:52:19.509 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:52:19.802 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:52:19.833 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:52:19.834 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:52:19.834 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:52:19.834 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:52:19.834 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:52:19.847 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:52:19.847 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:52:19.847 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:52:19.847 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:52:19.853 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:52:19.921 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:52:21.150 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:54:21.218 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:54:21.259 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:54:23.117 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:54:23.135 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:54:23.135 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:54:23.135 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:54:23.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:54:23.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:54:23.826 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:54:23.911 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:54:23.911 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:54:23.914 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:54:24.128 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:54:24.167 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:54:24.167 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:54:24.168 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:54:24.168 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:54:24.168 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:54:24.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:54:24.185 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:54:24.185 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:54:24.185 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:54:24.188 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:54:24.232 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:54:25.400 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:56:25.486 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:56:25.528 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:56:26.562 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:56:26.579 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:56:26.579 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:56:26.579 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:56:26.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:56:26.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:56:27.186 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:56:27.257 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:56:27.258 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:56:27.260 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:56:27.485 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:56:27.529 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:56:27.530 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:56:27.530 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:56:27.530 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:56:27.530 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:56:27.537 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:56:27.537 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:56:27.537 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:56:27.537 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:56:27.543 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:56:27.584 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:56:28.776 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:58:28.848 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T05:58:28.898 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:58:30.020 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T05:58:30.038 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T05:58:30.038 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T05:58:30.038 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:58:30.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:58:30.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T05:58:30.658 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T05:58:30.723 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T05:58:30.724 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T05:58:30.725 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:58:30.940 INFO:teuthology.misc:Compressing logs... 2021-03-23T05:58:30.989 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T05:58:30.989 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T05:58:30.990 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T05:58:30.990 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T05:58:30.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T05:58:30.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T05:58:30.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T05:58:30.994 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T05:58:30.995 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T05:58:30.999 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T05:58:31.038 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T05:58:32.124 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:00:32.196 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:00:32.241 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:00:33.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:00:33.385 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:00:33.386 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:00:33.386 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:00:33.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:00:33.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:00:34.003 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:00:34.071 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:00:34.072 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:00:34.074 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:00:34.252 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:00:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:00:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:00:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:00:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:00:34.303 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:00:34.346 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:00:35.541 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:02:35.624 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:02:35.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:02:37.710 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:02:37.728 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:02:37.729 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:02:37.729 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:02:37.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:02:38.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:02:38.355 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:02:38.422 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:02:38.422 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:02:38.424 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:02:38.636 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:02:38.687 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:02:38.687 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:02:38.687 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:02:38.687 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:02:38.692 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:02:38.736 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:02:39.893 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:04:39.966 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:04:40.008 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:04:41.137 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:04:41.154 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:04:41.155 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:04:41.155 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:04:41.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:04:41.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:04:41.763 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:04:41.832 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:04:41.832 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:04:41.833 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:04:42.018 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:04:42.057 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:04:42.057 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:04:42.057 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:04:42.057 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:04:42.057 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:04:42.066 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:04:42.066 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:04:42.066 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:04:42.067 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:04:42.074 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:04:42.116 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:04:43.172 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:06:43.244 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:06:43.289 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:06:44.923 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:06:44.941 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:06:44.941 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:06:44.941 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:06:44.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:06:45.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:06:45.575 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:06:45.648 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:06:45.648 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:06:45.651 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:06:45.832 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:06:45.881 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:06:45.881 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:06:45.881 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:06:45.881 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:06:45.886 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:06:45.929 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:06:47.174 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:08:47.263 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:08:47.387 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:08:49.187 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:08:49.204 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:08:49.204 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:08:49.204 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:08:49.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:08:49.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:08:49.820 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:08:49.893 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:08:49.893 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:08:49.895 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:08:50.084 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:08:50.131 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:08:50.131 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:08:50.131 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:08:50.131 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:08:50.145 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:08:50.146 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:08:50.147 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:08:50.147 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:08:50.147 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:08:50.147 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:08:50.187 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:08:51.231 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:10:51.309 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:10:51.350 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:10:53.207 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:10:53.224 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:10:53.224 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:10:53.224 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:10:53.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:10:53.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:10:53.823 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:10:53.889 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:10:53.889 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:10:53.891 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:10:54.119 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:10:54.160 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:10:54.160 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:10:54.160 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:10:54.160 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:10:54.163 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:10:54.216 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:10:55.307 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:12:55.373 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:12:55.413 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:12:56.567 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:12:56.584 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:12:56.585 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:12:56.585 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:12:56.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:12:56.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:12:57.198 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:12:57.273 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:12:57.273 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:12:57.275 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:12:57.459 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:12:57.504 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:12:57.505 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:12:57.505 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:12:57.505 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:12:57.510 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:12:57.551 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:12:58.606 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:14:58.678 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:14:58.723 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:14:59.848 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:14:59.865 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:14:59.866 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:14:59.866 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:14:59.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:15:00.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:15:00.463 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:15:00.530 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:15:00.531 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:15:00.533 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:15:00.757 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:15:00.794 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:15:00.794 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:15:00.794 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:15:00.794 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:15:00.794 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:15:00.810 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:15:00.810 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:15:00.810 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:15:00.810 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:15:00.817 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:15:00.855 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:15:01.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:17:01.936 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:17:01.978 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:17:03.867 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:17:03.884 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:17:03.885 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:17:03.885 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:17:03.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:17:04.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:17:04.482 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:17:04.552 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:17:04.552 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:17:04.554 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:17:04.714 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:17:04.761 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:17:04.761 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:17:04.761 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:17:04.761 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:17:04.761 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:17:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:17:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:17:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:17:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:17:04.778 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:17:04.823 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:17:05.970 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:19:06.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:19:06.093 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:19:08.144 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:19:08.162 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:19:08.163 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:19:08.163 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:19:08.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:19:08.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:19:08.787 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:19:08.860 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:19:08.860 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:19:08.862 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:19:09.060 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:19:09.105 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:19:09.105 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:19:09.105 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:19:09.106 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:19:09.115 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:19:09.159 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:19:10.769 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:21:10.835 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:21:10.879 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:21:12.851 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:21:12.871 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:21:12.872 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:21:12.872 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:21:12.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:21:13.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:21:13.495 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:21:13.562 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:21:13.562 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:21:13.564 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:21:13.751 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:21:13.793 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:21:13.794 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:21:13.794 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:21:13.794 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:21:13.794 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:21:13.797 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:21:13.797 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:21:13.798 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:21:13.798 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:21:13.802 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:21:13.844 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:21:15.418 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:23:15.496 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:23:15.537 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:23:17.462 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:23:17.479 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:23:17.480 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:23:17.480 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:23:17.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:23:17.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:23:18.091 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:23:18.158 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:23:18.158 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:23:18.160 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:23:18.320 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:23:18.375 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:23:18.375 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:23:18.375 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:23:18.376 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:23:18.376 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:23:18.392 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:23:18.392 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:23:18.392 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:23:18.392 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:23:18.399 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:23:18.447 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:23:19.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:25:19.904 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:25:19.949 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:25:22.250 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:25:22.274 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:25:22.274 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:25:22.274 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:25:22.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:25:22.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:25:22.898 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:25:22.965 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:25:22.966 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:25:22.968 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:25:23.168 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:25:23.213 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:25:23.213 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:25:23.213 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:25:23.213 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:25:23.220 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:25:23.260 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:25:25.778 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:27:25.849 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:27:25.894 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:27:28.036 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:27:28.054 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:27:28.054 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:27:28.055 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:27:28.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:27:28.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:27:28.688 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:27:28.757 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:27:28.757 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:27:28.759 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:27:28.958 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:27:29.005 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:27:29.005 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:27:29.005 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:27:29.006 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:27:29.012 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:27:29.054 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:27:30.214 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:29:30.286 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:29:30.330 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:29:32.546 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:29:32.564 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:29:32.564 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:29:32.564 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:29:32.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:29:32.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:29:33.183 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:29:33.248 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:29:33.248 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:29:33.250 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:29:33.422 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:29:33.461 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:29:33.461 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:29:33.461 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:29:33.461 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:29:33.461 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:29:33.473 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:29:33.474 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:29:33.474 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:29:33.474 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:29:33.478 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:29:33.526 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:29:34.669 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:31:34.742 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:31:34.794 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:31:36.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:31:36.222 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:31:36.222 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:31:36.222 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:31:36.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:31:36.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:31:36.864 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:31:36.936 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:31:36.936 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:31:36.939 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:31:37.126 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:31:37.174 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:31:37.174 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:31:37.174 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:31:37.175 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:31:37.180 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:31:37.223 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:31:38.334 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:33:38.403 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:33:38.579 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:33:39.749 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:33:39.768 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:33:39.769 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:33:39.769 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:33:39.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:33:40.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:33:40.403 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:33:40.489 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:33:40.489 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:33:40.492 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:33:40.662 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:33:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:33:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:33:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:33:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:33:40.718 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:33:40.718 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:33:40.718 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:33:40.718 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:33:40.718 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:33:40.719 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:33:40.765 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:33:41.942 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:35:42.017 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:35:42.068 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:35:43.276 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:35:43.313 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:35:43.314 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:35:43.314 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:35:43.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:35:43.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:35:43.959 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:35:44.029 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:35:44.030 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:35:44.031 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:35:44.211 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:35:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:35:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:35:44.263 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:35:44.263 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:35:44.263 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:35:44.307 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:35:45.432 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:37:45.513 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:37:45.565 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:37:46.947 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:37:46.971 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:37:46.971 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:37:46.971 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:37:46.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:37:47.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:37:47.598 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:37:47.670 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:37:47.670 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:37:47.672 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:37:47.845 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:37:47.889 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:37:47.889 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:37:47.889 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:37:47.890 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:37:47.895 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:37:47.942 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:37:49.046 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:39:49.116 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:39:49.178 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:39:50.494 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:39:50.518 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:39:50.519 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:39:50.519 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:39:50.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:39:50.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:39:51.208 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:39:51.282 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:39:51.282 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:39:51.284 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:39:51.471 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:39:51.517 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:39:51.518 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:39:51.518 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:39:51.518 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:39:51.518 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:39:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:39:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:39:51.540 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:39:51.540 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:39:51.545 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:39:51.585 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:39:52.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:41:52.897 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:41:52.959 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:41:54.254 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:41:54.272 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:41:54.272 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:41:54.272 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:41:54.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:41:54.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:41:54.916 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:41:54.985 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:41:54.985 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:41:54.986 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:41:55.171 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:41:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:41:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:41:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:41:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:41:55.222 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:41:55.352 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:41:56.524 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:43:56.602 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:43:56.668 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:43:58.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:43:58.994 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:43:58.995 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:43:58.995 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:43:58.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:43:59.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:43:59.720 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:43:59.787 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:43:59.787 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:43:59.789 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:44:00.008 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:44:00.061 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:44:00.062 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:44:00.062 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:44:00.062 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:44:00.070 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:44:00.138 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:44:02.745 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:46:02.817 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:46:02.862 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:46:04.942 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:46:04.959 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:46:04.959 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:46:04.959 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:46:04.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:46:05.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:46:05.638 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:46:05.716 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:46:05.716 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:46:05.719 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:46:05.900 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:46:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:46:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:46:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:46:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:46:05.957 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:46:06.008 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:46:07.171 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:48:07.273 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:48:07.316 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:48:09.294 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:48:09.317 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:48:09.318 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:48:09.318 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:48:09.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:48:09.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:48:10.011 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:48:10.084 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:48:10.084 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:48:10.086 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:48:10.286 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:48:10.330 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:48:10.331 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:48:10.331 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:48:10.331 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:48:10.331 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:48:10.340 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:48:10.340 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:48:10.340 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:48:10.340 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:48:10.347 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:48:10.393 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:48:11.555 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:50:11.630 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:50:11.671 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:50:12.766 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:50:12.783 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:50:12.784 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:50:12.784 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:50:12.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:50:13.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:50:13.460 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:50:13.525 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:50:13.526 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:50:13.527 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:50:13.717 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:50:13.768 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:50:13.769 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:50:13.769 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:50:13.769 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:50:13.773 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:50:13.812 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:50:14.864 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:52:14.934 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:52:14.981 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:52:16.046 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:52:16.065 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:52:16.065 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:52:16.065 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:52:16.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:52:16.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:52:16.823 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:52:16.926 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:52:16.926 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:52:16.928 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:52:17.205 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:52:17.245 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:52:17.246 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:52:17.246 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:52:17.246 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:52:17.246 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:52:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:52:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:52:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:52:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:52:17.259 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:52:17.299 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:52:18.331 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:54:18.401 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:54:18.443 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:54:20.006 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:54:20.023 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:54:20.024 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:54:20.024 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:54:20.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:54:20.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:54:20.678 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:54:20.756 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:54:20.756 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:54:20.760 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:54:21.041 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:54:21.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:54:21.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:54:21.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:54:21.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:54:21.075 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:54:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:54:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:54:21.088 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:54:21.088 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:54:21.092 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:54:21.134 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:54:22.209 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:56:22.275 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:56:22.319 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:56:24.164 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:56:24.182 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:56:24.182 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:56:24.182 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:56:24.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:56:24.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:56:24.881 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:56:24.946 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:56:24.946 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:56:24.949 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:56:25.131 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:56:25.170 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:56:25.171 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:56:25.171 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:56:25.171 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:56:25.171 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:56:25.193 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:56:25.193 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:56:25.194 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:56:25.194 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:56:25.200 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:56:25.247 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:56:26.382 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:58:26.447 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T06:58:26.497 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:58:28.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T06:58:28.288 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T06:58:28.288 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T06:58:28.288 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:58:28.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:58:28.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T06:58:28.967 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T06:58:29.038 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T06:58:29.038 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T06:58:29.040 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:58:29.218 INFO:teuthology.misc:Compressing logs... 2021-03-23T06:58:29.277 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T06:58:29.277 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T06:58:29.277 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T06:58:29.277 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T06:58:29.277 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T06:58:29.278 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T06:58:29.278 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T06:58:29.278 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T06:58:29.278 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T06:58:29.281 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T06:58:29.324 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T06:58:30.390 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:00:30.460 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:00:30.511 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:00:32.545 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:00:32.573 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:00:32.573 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:00:32.573 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:00:32.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:00:32.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:00:33.287 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:00:33.353 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:00:33.353 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:00:33.356 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:00:33.551 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:00:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:00:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:00:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:00:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:00:33.606 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:00:33.666 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:00:35.295 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:02:35.367 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:02:35.433 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:02:37.427 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:02:37.455 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:02:37.455 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:02:37.457 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:02:37.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:02:37.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:02:38.119 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:02:38.194 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:02:38.195 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:02:38.197 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:02:38.387 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:02:38.439 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:02:38.440 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:02:38.440 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:02:38.440 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:02:38.445 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:02:38.489 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:02:39.621 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:04:39.719 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:04:39.764 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:04:40.864 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:04:40.881 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:04:40.882 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:04:40.882 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:04:40.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:04:41.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:04:41.523 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:04:41.592 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:04:41.592 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:04:41.596 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:04:41.782 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:04:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:04:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:04:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:04:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:04:41.831 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:04:41.870 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:04:42.939 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:06:43.007 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:06:43.048 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:06:44.198 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:06:44.215 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:06:44.215 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:06:44.215 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:06:44.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:06:44.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:06:44.855 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:06:44.922 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:06:44.922 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:06:44.925 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:06:45.111 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:06:45.157 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:06:45.157 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:06:45.157 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:06:45.158 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:06:45.162 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:06:45.205 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:06:46.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:08:46.359 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:08:46.400 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:08:47.519 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:08:47.536 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:08:47.536 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:08:47.536 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:08:47.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:08:47.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:08:48.212 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:08:48.287 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:08:48.287 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:08:48.290 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:08:48.468 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:08:48.527 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:08:48.527 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:08:48.528 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:08:48.528 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:08:48.529 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T07:08:48.529 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T07:08:48.529 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T07:08:48.529 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T07:08:48.529 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T07:08:48.534 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:08:48.574 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:08:49.672 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:10:49.760 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:10:49.811 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:10:50.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:10:50.888 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:10:50.888 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:10:50.888 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:10:50.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:10:51.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:10:51.515 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:10:51.586 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:10:51.586 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:10:51.589 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:10:51.785 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:10:51.834 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:10:51.834 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:10:51.834 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:10:51.835 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:10:51.841 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:10:51.901 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:10:52.970 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:12:53.040 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:12:53.082 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:12:54.157 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:12:54.175 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:12:54.175 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:12:54.175 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:12:54.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:12:54.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:12:54.850 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:12:54.920 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:12:54.921 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:12:54.923 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:12:55.091 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:12:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:12:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:12:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:12:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:12:55.144 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:12:55.188 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:12:56.348 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:14:56.419 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:14:56.459 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:14:57.508 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:14:57.525 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:14:57.525 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:14:57.525 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:14:57.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:14:57.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:14:58.199 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:14:58.272 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:14:58.272 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:14:58.274 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:14:58.473 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:14:58.513 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:14:58.513 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:14:58.513 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:14:58.513 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:14:58.518 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:14:58.558 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:14:59.654 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:16:59.724 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:16:59.766 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:17:00.840 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:17:00.932 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:17:00.933 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:17:00.933 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:17:00.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:17:01.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:17:01.555 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:17:01.625 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:17:01.625 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:17:01.627 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:17:01.809 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:17:01.859 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:17:01.859 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:17:01.859 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:17:01.859 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:17:01.867 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:17:01.906 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:17:02.954 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:19:03.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:19:03.061 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:19:04.140 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:19:04.158 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:19:04.158 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:19:04.158 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:19:04.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:19:04.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:19:04.780 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:19:04.852 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:19:04.853 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:19:04.855 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:19:05.058 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:19:05.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:19:05.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:19:05.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:19:05.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:19:05.114 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:19:05.156 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:19:06.264 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:21:06.334 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:21:06.375 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:21:07.446 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:21:07.465 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:21:07.465 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:21:07.465 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:21:07.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:21:07.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:21:08.099 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:21:08.174 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:21:08.174 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:21:08.176 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:21:08.375 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:21:08.424 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:21:08.425 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:21:08.425 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:21:08.425 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:21:08.428 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T07:21:08.428 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T07:21:08.428 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T07:21:08.428 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T07:21:08.428 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T07:21:08.430 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:21:08.469 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:21:09.538 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:23:09.608 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:23:09.649 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:23:10.737 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:23:10.759 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:23:10.759 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:23:10.760 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:23:10.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:23:11.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:23:11.383 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:23:11.459 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:23:11.459 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:23:11.461 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:23:11.663 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:23:11.711 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:23:11.711 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:23:11.711 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:23:11.711 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:23:11.718 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:23:11.757 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:23:12.863 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:25:12.940 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:25:12.985 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:25:14.017 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:25:14.035 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:25:14.035 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:25:14.035 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:25:14.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:25:14.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:25:14.655 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:25:14.728 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:25:14.728 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:25:14.730 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:25:14.918 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:25:14.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:25:14.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:25:14.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:25:14.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:25:14.976 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:25:15.018 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:25:16.072 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:27:16.154 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:27:16.197 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:27:17.335 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:27:17.352 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:27:17.352 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:27:17.353 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:27:17.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:27:17.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:27:18.003 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:27:18.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:27:18.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:27:18.070 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:27:18.278 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:27:18.319 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T07:27:18.319 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T07:27:18.319 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T07:27:18.319 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T07:27:18.319 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T07:27:18.341 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:27:18.341 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:27:18.341 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:27:18.342 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:27:18.347 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:27:18.396 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:27:19.484 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:29:19.553 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:29:19.595 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:29:20.674 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:29:20.696 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:29:20.697 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:29:20.697 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:29:20.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:29:20.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:29:21.319 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:29:21.393 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:29:21.393 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:29:21.395 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:29:21.596 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:29:21.645 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:29:21.646 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:29:21.646 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:29:21.646 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:29:21.654 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:29:21.654 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T07:29:21.655 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T07:29:21.655 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T07:29:21.655 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T07:29:21.655 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T07:29:21.693 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:29:22.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:31:22.798 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:31:22.838 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:31:23.904 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:31:23.942 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:31:23.943 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:31:23.943 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:31:23.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:31:24.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:31:24.574 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:31:24.641 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:31:24.641 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:31:24.644 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:31:24.822 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:31:24.864 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:31:24.865 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:31:24.865 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:31:24.865 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:31:24.870 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:31:24.912 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:31:26.034 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:33:26.100 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:33:26.152 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:33:27.230 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:33:27.248 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:33:27.248 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:33:27.248 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:33:27.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:33:27.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:33:27.879 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:33:27.949 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:33:27.949 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:33:27.952 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:33:28.151 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:33:28.199 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:33:28.199 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:33:28.199 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:33:28.199 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:33:28.208 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:33:28.209 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T07:33:28.209 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T07:33:28.210 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T07:33:28.210 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T07:33:28.210 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T07:33:28.252 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:33:29.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:35:29.422 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:35:29.465 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:35:30.601 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:35:30.623 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:35:30.623 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:35:30.623 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:35:30.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:35:30.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:35:31.238 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:35:31.304 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:35:31.304 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:35:31.307 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:35:31.483 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:35:31.526 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:35:31.527 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:35:31.527 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:35:31.527 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:35:31.531 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:35:31.568 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:35:32.559 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:37:32.624 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:37:32.664 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:37:33.679 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:37:33.696 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:37:33.696 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:37:33.696 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:37:33.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:37:33.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:37:34.311 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:37:34.380 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:37:34.381 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:37:34.383 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:37:34.592 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:37:34.640 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:37:34.640 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:37:34.640 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:37:34.640 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:37:34.649 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:37:34.688 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:37:35.673 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:39:35.740 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:39:35.783 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:39:36.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:39:36.829 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:39:36.830 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:39:36.830 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:39:36.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:39:37.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:39:37.459 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:39:37.528 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:39:37.529 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:39:37.531 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:39:37.726 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:39:37.775 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:39:37.775 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:39:37.776 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:39:37.776 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:39:37.784 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:39:37.823 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:39:38.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:41:38.925 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:41:38.965 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:41:40.156 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:41:40.178 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:41:40.178 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:41:40.178 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:41:40.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:41:40.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:41:40.820 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:41:40.887 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:41:40.887 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:41:40.889 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:41:41.090 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:41:41.144 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:41:41.144 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:41:41.144 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:41:41.144 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:41:41.148 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T07:41:41.148 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T07:41:41.148 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T07:41:41.148 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T07:41:41.148 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T07:41:41.151 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:41:41.191 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:41:42.343 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:43:42.430 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:43:42.471 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:43:43.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:43:43.617 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:43:43.618 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:43:43.618 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:43:43.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:43:43.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:43:44.239 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:43:44.310 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:43:44.310 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:43:44.312 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:43:44.519 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:43:44.565 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:43:44.565 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:43:44.566 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:43:44.566 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:43:44.569 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:43:44.612 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:43:45.678 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:45:45.746 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:45:45.788 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:45:46.857 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:45:46.875 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:45:46.875 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:45:46.875 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:45:46.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:45:47.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:45:47.506 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:45:47.580 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:45:47.580 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:45:47.583 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:45:47.774 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:45:47.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T07:45:47.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T07:45:47.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T07:45:47.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T07:45:47.812 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T07:45:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:45:47.820 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:45:47.820 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:45:47.820 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:45:47.826 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:45:47.865 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:45:48.912 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:47:48.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:47:49.020 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:47:50.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:47:50.069 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:47:50.069 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:47:50.069 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:47:50.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:47:50.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:47:50.699 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:47:50.767 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:47:50.767 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:47:50.770 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:47:50.984 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:47:51.030 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:47:51.030 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:47:51.031 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:47:51.031 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:47:51.036 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:47:51.072 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:47:52.086 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:49:52.153 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:49:52.190 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:49:53.192 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:49:53.209 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:49:53.209 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:49:53.210 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:49:53.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:49:53.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:49:53.822 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:49:53.891 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:49:53.891 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:49:53.892 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:49:54.103 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:49:54.150 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:49:54.150 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:49:54.150 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:49:54.150 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:49:54.157 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:49:54.196 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:49:55.206 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:51:55.271 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:51:55.392 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:51:56.499 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:51:56.516 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:51:56.516 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:51:56.516 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:51:56.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:51:56.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:51:57.138 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:51:57.208 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:51:57.208 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:51:57.211 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:51:57.392 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:51:57.435 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:51:57.435 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:51:57.435 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:51:57.435 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:51:57.439 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:51:57.475 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:51:58.478 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:53:58.544 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:53:58.583 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:53:59.644 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:53:59.660 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:53:59.661 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:53:59.661 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:53:59.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:53:59.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:54:00.350 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:54:00.418 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:54:00.418 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:54:00.420 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:54:00.624 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:54:00.662 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T07:54:00.662 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T07:54:00.662 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T07:54:00.663 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T07:54:00.663 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T07:54:00.672 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:54:00.673 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:54:00.673 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:54:00.673 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:54:00.678 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:54:00.720 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:54:01.757 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:56:01.830 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:56:01.872 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:56:02.917 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:56:02.935 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:56:02.935 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:56:02.936 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:56:02.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:56:03.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:56:03.554 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:56:03.618 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:56:03.618 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:56:03.620 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:56:03.796 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:56:03.842 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:56:03.842 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:56:03.842 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:56:03.842 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:56:03.847 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:56:03.887 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:56:04.934 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:58:05.007 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T07:58:05.049 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:58:06.119 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T07:58:06.136 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T07:58:06.136 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T07:58:06.136 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:58:06.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:58:06.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T07:58:06.736 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T07:58:06.801 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T07:58:06.801 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T07:58:06.803 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:58:06.984 INFO:teuthology.misc:Compressing logs... 2021-03-23T07:58:07.028 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T07:58:07.029 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T07:58:07.029 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T07:58:07.029 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T07:58:07.034 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T07:58:07.074 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T07:58:08.137 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:00:08.216 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:00:08.260 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:00:09.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:00:09.449 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:00:09.449 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:00:09.449 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:00:09.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:00:09.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:00:10.170 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:00:10.238 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:00:10.239 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:00:10.241 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:00:10.428 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:00:10.463 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T08:00:10.464 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T08:00:10.464 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T08:00:10.464 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T08:00:10.464 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T08:00:10.482 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:00:10.482 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:00:10.483 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:00:10.483 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:00:10.489 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:00:10.534 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:00:11.714 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:02:11.791 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:02:11.861 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:02:12.978 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:02:12.996 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:02:12.996 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:02:12.996 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:02:12.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:02:13.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:02:13.630 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:02:13.702 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:02:13.702 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:02:13.704 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:02:13.900 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:02:13.945 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:02:13.945 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:02:13.945 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:02:13.945 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:02:13.949 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:02:13.988 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:02:15.004 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:04:15.079 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:04:15.129 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:04:16.182 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:04:16.206 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:04:16.206 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:04:16.206 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:04:16.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:04:16.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:04:16.823 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:04:16.883 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:04:16.883 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:04:16.886 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:04:17.090 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:04:17.136 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:04:17.136 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:04:17.136 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:04:17.136 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:04:17.141 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:04:17.181 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:04:18.253 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:06:18.322 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:06:18.363 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:06:19.390 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:06:19.406 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:06:19.407 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:06:19.407 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:06:19.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:06:19.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:06:20.066 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:06:20.141 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:06:20.141 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:06:20.142 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:06:20.316 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:06:20.364 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T08:06:20.364 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T08:06:20.364 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T08:06:20.364 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T08:06:20.364 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T08:06:20.365 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:06:20.365 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:06:20.366 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:06:20.366 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:06:20.370 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:06:20.410 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:06:21.444 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:08:21.511 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:08:21.550 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:08:22.619 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:08:22.636 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:08:22.636 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:08:22.637 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:08:22.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:08:22.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:08:23.292 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:08:23.361 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:08:23.361 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:08:23.364 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:08:23.539 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:08:23.589 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:08:23.589 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:08:23.589 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:08:23.589 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:08:23.595 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T08:08:23.596 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T08:08:23.596 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T08:08:23.596 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T08:08:23.596 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T08:08:23.597 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:08:23.635 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:08:24.629 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:10:24.695 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:10:24.737 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:10:25.735 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:10:25.751 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:10:25.752 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:10:25.752 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:10:25.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:10:26.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:10:26.389 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:10:26.456 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:10:26.456 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:10:26.459 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:10:26.646 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:10:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:10:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:10:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:10:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:10:26.695 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:10:26.733 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:10:27.674 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:12:27.739 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:12:27.784 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:12:28.833 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:12:28.868 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:12:28.868 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:12:28.869 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:12:28.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:12:29.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:12:29.499 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:12:29.566 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:12:29.566 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:12:29.568 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:12:29.762 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:12:29.809 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:12:29.809 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:12:29.810 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:12:29.810 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:12:29.820 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:12:29.859 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:12:30.885 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:14:30.955 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:14:30.994 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:14:32.013 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:14:32.031 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:14:32.031 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:14:32.031 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:14:32.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:14:32.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:14:32.659 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:14:32.732 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:14:32.732 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:14:32.734 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:14:32.924 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:14:32.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:14:32.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:14:32.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:14:32.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:14:32.974 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:14:33.014 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:14:34.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:16:34.076 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:16:34.118 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:16:35.291 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:16:35.308 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:16:35.308 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:16:35.308 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:16:35.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:16:35.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:16:35.952 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:16:36.016 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:16:36.016 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:16:36.019 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:16:36.203 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:16:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:16:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:16:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:16:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:16:36.256 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:16:36.300 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:16:37.360 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:18:37.428 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:18:37.468 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:18:39.315 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:18:39.335 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:18:39.335 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:18:39.335 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:18:39.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:18:39.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:18:39.955 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:18:40.017 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:18:40.018 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:18:40.020 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:18:40.201 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:18:40.248 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:18:40.248 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:18:40.248 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:18:40.248 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:18:40.253 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:18:40.294 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:18:41.333 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:20:41.407 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:20:41.456 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:20:42.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:20:42.518 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:20:42.518 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:20:42.518 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:20:42.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:20:42.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:20:43.181 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:20:43.249 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:20:43.250 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:20:43.253 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:20:43.425 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:20:43.470 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:20:43.470 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:20:43.471 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:20:43.471 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:20:43.475 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:20:43.512 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:20:44.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:22:44.607 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:22:44.648 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:22:45.636 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:22:45.655 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:22:45.655 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:22:45.655 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:22:45.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:22:45.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:22:46.287 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:22:46.354 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:22:46.354 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:22:46.356 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:22:46.552 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:22:46.596 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:22:46.596 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:22:46.596 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:22:46.596 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:22:46.603 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:22:46.641 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:22:47.641 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:24:47.714 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:24:47.755 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:24:48.801 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:24:48.818 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:24:48.818 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:24:48.818 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:24:48.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:24:49.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:24:49.494 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:24:49.562 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:24:49.562 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:24:49.564 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:24:49.770 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:24:49.805 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T08:24:49.805 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T08:24:49.805 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T08:24:49.806 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T08:24:49.806 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T08:24:49.824 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:24:49.824 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:24:49.825 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:24:49.825 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:24:49.831 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:24:49.945 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:24:50.981 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:26:51.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:26:51.085 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:26:52.155 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:26:52.173 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:26:52.173 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:26:52.173 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:26:52.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:26:52.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:26:52.786 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:26:52.852 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:26:52.853 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:26:52.855 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:26:53.066 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:26:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:26:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:26:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:26:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:26:53.132 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:26:53.133 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T08:26:53.133 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T08:26:53.134 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T08:26:53.134 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T08:26:53.134 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T08:26:53.172 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:26:54.212 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:28:54.281 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:28:54.322 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:28:55.399 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:28:55.417 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:28:55.417 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:28:55.417 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:28:55.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:28:55.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:28:56.039 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:28:56.119 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:28:56.120 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:28:56.121 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:28:56.334 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:28:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:28:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:28:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:28:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:28:56.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:28:56.426 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:28:57.523 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:30:57.602 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:30:57.642 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:30:58.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:30:58.710 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:30:58.710 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:30:58.711 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:30:58.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:30:58.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:30:59.382 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:30:59.448 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:30:59.448 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:30:59.450 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:30:59.628 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:30:59.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:30:59.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:30:59.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:30:59.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:30:59.681 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:30:59.722 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:31:00.777 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:33:00.841 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:33:00.882 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:33:01.981 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:33:01.998 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:33:01.998 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:33:01.998 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:33:01.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:33:02.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:33:02.619 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:33:02.686 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:33:02.687 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:33:02.688 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:33:02.912 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:33:02.955 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:33:02.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:33:02.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:33:02.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:33:02.961 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:33:02.999 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:33:04.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:35:04.107 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:35:04.150 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:35:05.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:35:05.214 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:35:05.215 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:35:05.215 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:35:05.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:35:05.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:35:05.834 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:35:05.905 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:35:05.905 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:35:05.908 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:35:06.105 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:35:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:35:06.152 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:35:06.152 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:35:06.152 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:35:06.161 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:35:06.204 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:35:07.266 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:37:07.330 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:37:07.377 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:37:09.369 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:37:09.387 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:37:09.387 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:37:09.387 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:37:09.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:37:09.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:37:10.004 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:37:10.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:37:10.069 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:37:10.073 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:37:10.288 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:37:10.333 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:37:10.334 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:37:10.334 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:37:10.334 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:37:10.341 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:37:10.408 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:37:11.513 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:39:11.584 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:39:11.626 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:39:12.664 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:39:12.682 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:39:12.683 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:39:12.683 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:39:12.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:39:13.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:39:13.393 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:39:13.454 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:39:13.454 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:39:13.456 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:39:13.634 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:39:13.681 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:39:13.681 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:39:13.681 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:39:13.681 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:39:13.686 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:39:13.726 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:39:14.787 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:41:14.857 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:41:14.909 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:41:16.984 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:41:17.001 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:41:17.002 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:41:17.002 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:41:17.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:41:17.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:41:17.683 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:41:17.753 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:41:17.753 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:41:17.756 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:41:17.956 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:41:18.000 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T08:41:18.000 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T08:41:18.001 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T08:41:18.001 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T08:41:18.001 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T08:41:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:41:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:41:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:41:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:41:18.031 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:41:18.073 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:41:19.118 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:43:19.183 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:43:19.223 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:43:21.111 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:43:21.128 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:43:21.129 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:43:21.129 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:43:21.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:43:21.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:43:21.850 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:43:21.917 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:43:21.917 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:43:21.920 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:43:22.124 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:43:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:43:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:43:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:43:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:43:22.177 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:43:22.221 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:43:23.776 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:45:23.839 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:45:23.892 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:45:26.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:45:26.042 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:45:26.042 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:45:26.042 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:45:26.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:45:26.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:45:26.688 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:45:26.755 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:45:26.755 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:45:26.756 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:45:26.954 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:45:27.005 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:45:27.006 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:45:27.006 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:45:27.006 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:45:27.012 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T08:45:27.012 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T08:45:27.012 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T08:45:27.012 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T08:45:27.013 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T08:45:27.019 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:45:27.073 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:45:29.043 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:47:29.109 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:47:29.155 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:47:31.066 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:47:31.087 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:47:31.087 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:47:31.087 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:47:31.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:47:31.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:47:31.712 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:47:31.784 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:47:31.784 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:47:31.785 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:47:31.973 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:47:32.023 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:47:32.023 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:47:32.023 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:47:32.023 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:47:32.032 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T08:47:32.033 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T08:47:32.033 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T08:47:32.033 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T08:47:32.033 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T08:47:32.041 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:47:32.090 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:47:33.221 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:49:33.290 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:49:33.330 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:49:34.426 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:49:34.449 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:49:34.449 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:49:34.449 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:49:34.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:49:34.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:49:35.126 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:49:35.198 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:49:35.198 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:49:35.200 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:49:35.383 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:49:35.426 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T08:49:35.426 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T08:49:35.426 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T08:49:35.426 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T08:49:35.426 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T08:49:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:49:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:49:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:49:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:49:35.455 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:49:35.497 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:49:36.635 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:51:36.713 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:51:36.753 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:51:37.792 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:51:37.809 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:51:37.809 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:51:37.809 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:51:37.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:51:38.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:51:38.474 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:51:38.543 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:51:38.543 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:51:38.547 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:51:38.718 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:51:38.763 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:51:38.764 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:51:38.764 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:51:38.764 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:51:38.770 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:51:38.809 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:51:39.848 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:53:39.918 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:53:39.961 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:53:41.831 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:53:41.850 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:53:41.850 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:53:41.850 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:53:41.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:53:42.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:53:42.491 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:53:42.561 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:53:42.562 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:53:42.564 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:53:42.755 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:53:42.801 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:53:42.801 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:53:42.802 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:53:42.802 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:53:42.809 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:53:42.849 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:53:43.936 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:55:44.015 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:55:44.055 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:55:45.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:55:45.088 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:55:45.089 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:55:45.089 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:55:45.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:55:45.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:55:45.717 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:55:45.788 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:55:45.788 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:55:45.791 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:55:45.983 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:55:46.022 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:55:46.022 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:55:46.022 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:55:46.023 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:55:46.028 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:55:46.066 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:55:47.035 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:57:47.103 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:57:47.145 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:57:48.200 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:57:48.218 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:57:48.218 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:57:48.218 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:57:48.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:57:48.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:57:48.844 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:57:48.918 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:57:48.918 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:57:48.921 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:57:49.118 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:57:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:57:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:57:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:57:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:57:49.169 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:57:49.208 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:57:50.270 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:59:50.341 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T08:59:50.460 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:59:51.592 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T08:59:51.611 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T08:59:51.612 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T08:59:51.612 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:59:51.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:59:51.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T08:59:52.256 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T08:59:52.318 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T08:59:52.319 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T08:59:52.321 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:59:52.518 INFO:teuthology.misc:Compressing logs... 2021-03-23T08:59:52.560 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T08:59:52.560 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T08:59:52.560 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T08:59:52.561 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T08:59:52.565 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T08:59:52.606 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T08:59:53.697 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:01:53.767 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:01:53.810 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:01:54.874 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:01:54.892 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:01:54.892 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:01:54.892 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:01:54.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:01:55.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:01:55.528 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:01:55.593 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:01:55.593 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:01:55.595 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:01:55.788 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:01:55.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:01:55.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:01:55.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:01:55.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:01:55.843 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:01:55.901 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:01:57.007 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:03:57.076 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:03:57.117 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:03:58.207 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:03:58.225 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:03:58.226 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:03:58.226 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:03:58.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:03:58.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:03:58.851 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:03:58.921 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:03:58.922 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:03:58.924 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:03:59.122 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:03:59.162 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:03:59.163 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:03:59.163 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:03:59.163 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:03:59.168 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:03:59.207 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:04:00.265 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:06:00.339 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:06:00.382 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:06:01.443 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:06:01.461 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:06:01.461 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:06:01.461 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:06:01.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:06:01.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:06:02.100 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:06:02.167 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:06:02.167 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:06:02.170 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:06:02.377 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:06:02.428 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T09:06:02.429 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T09:06:02.429 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T09:06:02.429 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T09:06:02.429 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T09:06:02.430 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:06:02.431 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:06:02.431 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:06:02.431 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:06:02.436 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:06:02.476 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:06:03.646 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:08:03.720 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:08:03.761 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:08:04.774 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:08:04.792 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:08:04.792 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:08:04.792 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:08:04.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:08:05.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:08:05.398 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:08:05.469 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:08:05.470 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:08:05.472 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:08:05.650 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:08:05.699 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:08:05.699 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:08:05.699 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:08:05.699 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:08:05.703 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:08:05.743 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:08:06.781 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:10:06.862 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:10:06.903 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:10:07.932 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:10:07.949 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:10:07.950 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:10:07.950 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:10:07.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:10:08.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:10:08.574 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:10:08.645 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:10:08.645 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:10:08.647 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:10:08.837 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:10:08.888 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:10:08.888 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:10:08.889 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:10:08.889 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:10:08.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T09:10:08.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T09:10:08.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T09:10:08.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T09:10:08.896 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T09:10:08.901 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:10:08.938 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:10:09.924 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:12:09.990 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:12:10.032 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:12:11.061 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:12:11.079 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:12:11.079 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:12:11.079 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:12:11.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:12:11.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:12:11.761 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:12:11.832 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:12:11.832 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:12:11.834 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:12:12.031 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:12:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:12:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:12:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:12:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:12:12.083 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:12:12.124 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:12:13.198 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:14:13.268 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:14:13.307 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:14:14.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:14:14.385 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:14:14.385 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:14:14.385 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:14:14.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:14:14.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:14:15.014 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:14:15.080 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:14:15.080 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:14:15.083 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:14:15.279 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:14:15.345 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:14:15.345 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:14:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:14:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:14:15.346 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:14:15.347 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T09:14:15.348 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T09:14:15.348 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T09:14:15.348 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T09:14:15.348 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T09:14:15.398 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:14:16.483 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:16:16.554 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:16:16.595 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:16:17.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:16:17.713 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:16:17.713 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:16:17.713 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:16:17.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:16:18.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:16:18.390 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:16:18.460 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:16:18.460 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:16:18.463 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:16:18.687 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:16:18.724 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T09:16:18.724 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T09:16:18.724 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T09:16:18.724 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T09:16:18.725 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T09:16:18.734 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:16:18.734 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:16:18.735 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:16:18.735 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:16:18.740 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:16:18.780 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:16:19.853 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:18:19.933 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:18:19.977 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:18:20.999 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:18:21.016 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:18:21.016 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:18:21.017 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:18:21.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:18:21.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:18:21.686 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:18:21.753 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:18:21.754 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:18:21.756 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:18:21.930 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:18:21.977 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:18:21.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:18:21.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:18:21.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:18:21.984 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:18:22.022 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:18:23.074 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:20:23.141 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:20:23.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:20:24.278 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:20:24.296 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:20:24.296 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:20:24.296 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:20:24.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:20:24.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:20:24.923 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:20:24.992 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:20:24.992 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:20:24.995 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:20:25.194 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:20:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:20:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:20:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:20:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:20:25.246 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:20:25.288 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:20:26.334 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:22:26.402 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:22:26.442 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:22:27.534 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:22:27.551 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:22:27.551 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:22:27.551 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:22:27.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:22:27.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:22:28.187 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:22:28.257 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:22:28.258 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:22:28.260 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:22:28.447 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:22:28.493 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:22:28.493 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:22:28.494 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:22:28.494 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:22:28.503 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:22:28.556 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:22:29.710 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:24:29.774 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:24:29.815 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:24:30.906 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:24:30.923 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:24:30.923 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:24:30.924 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:24:30.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:24:31.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:24:31.550 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:24:31.617 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:24:31.617 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:24:31.619 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:24:31.822 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:24:31.869 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:24:31.869 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:24:31.869 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:24:31.870 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:24:31.875 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:24:31.918 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:24:32.949 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:26:33.022 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:26:33.062 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:26:34.151 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:26:34.168 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:26:34.169 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:26:34.169 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:26:34.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:26:34.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:26:34.797 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:26:34.865 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:26:34.866 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:26:34.868 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:26:35.067 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:26:35.117 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:26:35.117 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:26:35.117 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:26:35.117 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:26:35.123 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:26:35.164 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:26:36.237 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:28:36.315 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:28:36.354 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:28:37.451 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:28:37.476 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:28:37.476 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:28:37.476 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:28:37.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:28:37.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:28:38.154 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:28:38.226 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:28:38.227 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:28:38.229 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:28:38.408 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:28:38.459 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:28:38.459 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:28:38.459 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:28:38.459 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:28:38.464 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:28:38.503 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:28:39.544 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:30:39.611 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:30:39.658 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:30:41.969 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:30:42.003 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:30:42.003 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:30:42.003 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:30:42.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:30:42.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:30:42.661 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:30:42.727 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:30:42.727 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:30:42.732 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:30:42.940 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:30:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:30:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:30:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:30:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:30:43.000 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:30:43.048 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:30:44.228 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:32:44.294 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:32:44.335 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:32:45.392 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:32:45.409 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:32:45.410 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:32:45.410 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:32:45.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:32:45.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:32:46.033 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:32:46.105 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:32:46.105 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:32:46.107 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:32:46.307 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:32:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:32:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:32:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:32:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:32:46.360 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:32:46.473 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:32:47.514 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:34:47.582 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:34:47.625 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:34:48.698 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:34:48.716 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:34:48.716 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:34:48.716 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:34:48.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:34:49.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:34:49.372 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:34:49.439 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:34:49.439 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:34:49.440 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:34:49.630 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:34:49.680 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T09:34:49.680 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T09:34:49.680 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T09:34:49.680 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T09:34:49.680 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T09:34:49.684 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:34:49.684 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:34:49.684 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:34:49.684 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:34:49.689 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:34:49.730 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:34:50.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:36:50.856 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:36:50.899 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:36:52.653 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:36:52.675 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:36:52.676 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:36:52.676 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:36:52.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:36:52.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:36:53.306 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:36:53.375 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:36:53.375 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:36:53.378 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:36:53.575 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:36:53.616 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:36:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:36:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:36:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:36:53.621 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:36:53.663 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:36:54.710 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:38:54.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:38:54.839 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:38:55.932 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:38:55.951 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:38:55.951 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:38:55.951 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:38:55.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:38:56.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:38:56.567 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:38:56.637 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:38:56.637 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:38:56.639 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:38:56.837 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:38:56.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:38:56.888 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:38:56.888 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:38:56.888 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:38:56.897 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:38:56.897 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T09:38:56.897 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T09:38:56.898 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T09:38:56.898 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T09:38:56.898 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T09:38:56.970 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:38:58.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:40:58.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:40:58.137 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:40:59.210 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:40:59.227 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:40:59.228 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:40:59.228 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:40:59.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:40:59.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:40:59.846 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:40:59.920 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:40:59.920 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:40:59.922 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:41:00.121 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:41:00.165 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:41:00.166 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:41:00.166 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:41:00.166 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:41:00.170 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:41:00.237 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:41:01.315 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:43:01.380 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:43:01.422 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:43:02.459 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:43:02.480 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:43:02.480 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:43:02.480 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:43:02.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:43:02.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:43:03.102 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:43:03.173 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:43:03.173 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:43:03.175 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:43:03.388 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:43:03.448 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:43:03.448 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:43:03.448 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:43:03.449 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:43:03.449 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:43:03.489 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:43:04.509 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:45:04.574 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:45:04.614 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:45:05.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:45:05.705 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:45:05.706 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:45:05.706 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:45:05.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:45:05.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:45:06.320 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:45:06.384 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:45:06.385 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:45:06.387 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:45:06.599 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:45:06.650 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:45:06.650 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:45:06.650 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:45:06.650 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:45:06.655 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T09:45:06.655 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T09:45:06.655 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T09:45:06.655 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T09:45:06.655 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T09:45:06.663 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:45:06.704 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:45:07.768 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:47:07.834 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:47:07.879 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:47:08.914 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:47:08.931 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:47:08.931 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:47:08.931 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:47:08.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:47:09.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:47:09.534 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:47:09.600 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:47:09.600 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:47:09.603 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:47:09.825 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:47:09.877 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:47:09.877 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:47:09.877 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:47:09.877 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:47:09.878 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:47:09.917 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:47:10.991 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:49:11.060 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:49:11.111 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:49:12.234 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:49:12.255 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:49:12.256 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:49:12.256 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:49:12.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:49:12.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:49:12.871 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:49:12.943 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:49:12.943 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:49:12.945 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:49:13.143 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:49:13.196 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:49:13.196 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:49:13.196 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:49:13.196 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:49:13.196 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:49:13.256 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:49:14.322 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:51:14.391 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:51:14.534 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:51:15.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:51:15.792 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:51:15.792 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:51:15.792 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:51:15.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:51:16.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:51:16.487 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:51:16.566 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:51:16.566 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:51:16.569 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:51:16.745 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:51:16.793 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:51:16.793 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:51:16.794 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:51:16.794 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:51:16.799 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:51:16.840 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:51:17.866 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:53:17.933 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:53:17.972 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:53:18.989 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:53:19.007 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:53:19.007 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:53:19.007 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:53:19.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:53:19.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:53:19.634 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:53:19.724 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:53:19.725 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:53:19.729 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:53:20.067 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:53:20.105 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T09:53:20.106 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T09:53:20.106 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T09:53:20.106 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T09:53:20.106 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T09:53:20.126 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:53:20.127 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:53:20.127 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:53:20.127 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:53:20.131 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:53:20.175 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:53:21.205 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:55:21.272 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:55:21.311 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:55:22.355 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:55:22.377 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:55:22.377 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:55:22.377 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:55:22.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:55:22.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:55:22.991 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:55:23.059 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:55:23.060 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:55:23.062 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:55:23.251 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:55:23.308 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:55:23.309 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:55:23.309 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:55:23.309 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:55:23.309 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:55:23.351 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:55:24.439 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:57:24.517 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:57:24.568 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:57:26.512 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:57:26.530 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:57:26.530 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:57:26.530 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:57:26.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:57:26.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:57:27.156 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:57:27.232 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:57:27.233 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:57:27.243 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:57:27.724 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:57:27.780 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T09:57:27.780 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T09:57:27.780 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T09:57:27.780 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T09:57:27.780 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T09:57:27.787 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:57:27.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:57:27.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:57:27.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:57:27.788 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:57:27.832 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:57:29.589 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:59:29.665 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T09:59:29.709 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:59:31.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T09:59:31.518 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T09:59:31.518 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T09:59:31.519 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:59:31.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:59:31.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T09:59:32.147 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T09:59:32.221 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T09:59:32.221 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T09:59:32.224 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:59:32.425 INFO:teuthology.misc:Compressing logs... 2021-03-23T09:59:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T09:59:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T09:59:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T09:59:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T09:59:32.473 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T09:59:32.514 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T09:59:33.576 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:01:33.640 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:01:33.683 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:01:34.754 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:01:34.777 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:01:34.778 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:01:34.778 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:01:34.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:01:35.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:01:35.505 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:01:35.581 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:01:35.581 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:01:35.584 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:01:35.840 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:01:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:01:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:01:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:01:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:01:35.896 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:01:35.937 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:01:37.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:03:37.087 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:03:37.126 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:03:38.171 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:03:38.189 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:03:38.189 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:03:38.189 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:03:38.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:03:38.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:03:38.798 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:03:38.870 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:03:38.870 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:03:38.872 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:03:39.142 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:03:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:03:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:03:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:03:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:03:39.197 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:03:39.241 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:03:40.284 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:05:40.353 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:05:40.422 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:05:42.337 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:05:42.360 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:05:42.360 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:05:42.360 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:05:42.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:05:42.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:05:42.987 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:05:43.066 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:05:43.067 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:05:43.069 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:05:43.254 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:05:43.294 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:05:43.294 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:05:43.294 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:05:43.294 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:05:43.298 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:05:43.339 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:05:44.840 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:07:44.909 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:07:44.947 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:07:46.922 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:07:46.945 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:07:46.945 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:07:46.945 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:07:46.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:07:47.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:07:47.567 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:07:47.634 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:07:47.634 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:07:47.637 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:07:47.822 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:07:47.867 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:07:47.867 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:07:47.867 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:07:47.867 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:07:47.873 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:07:47.984 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:07:49.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:09:50.042 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:09:50.083 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:09:52.070 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:09:52.090 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:09:52.090 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:09:52.090 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:09:52.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:09:52.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:09:52.749 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:09:52.816 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:09:52.817 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:09:52.819 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:09:53.018 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:09:53.069 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:09:53.070 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:09:53.070 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:09:53.070 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:09:53.076 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:09:53.118 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:09:54.273 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:11:54.347 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:11:54.388 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:11:56.290 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:11:56.309 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:11:56.309 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:11:56.310 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:11:56.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:11:56.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:11:56.935 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:11:57.007 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:11:57.008 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:11:57.010 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:11:57.207 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:11:57.259 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:11:57.260 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:11:57.260 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:11:57.260 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:11:57.269 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:11:57.310 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:11:58.577 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:13:58.642 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:13:58.685 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:14:00.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:14:00.609 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:14:00.609 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:14:00.609 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:14:00.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:14:00.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:14:01.263 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:14:01.330 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:14:01.330 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:14:01.333 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:14:01.501 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:14:01.544 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:14:01.544 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:14:01.544 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:14:01.545 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:14:01.550 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:14:01.589 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:14:03.151 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:16:03.222 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:16:03.288 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:16:05.262 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:16:05.279 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:16:05.279 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:16:05.280 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:16:05.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:16:05.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:16:05.897 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:16:05.966 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:16:05.966 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:16:05.969 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:16:06.175 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:16:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:16:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:16:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:16:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:16:06.231 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:16:06.274 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:16:08.173 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:18:08.248 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:18:08.290 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:18:10.313 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:18:10.331 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:18:10.331 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:18:10.332 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:18:10.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:18:10.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:18:11.006 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:18:11.082 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:18:11.082 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:18:11.084 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:18:11.256 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:18:11.310 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:18:11.310 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:18:11.311 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:18:11.311 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:18:11.317 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:18:11.357 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:18:12.955 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:20:13.023 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:20:13.083 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:20:15.202 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:20:15.219 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:20:15.220 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:20:15.220 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:20:15.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:20:15.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:20:15.839 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:20:15.912 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:20:15.912 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:20:15.914 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:20:16.106 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:20:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:20:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:20:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:20:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:20:16.157 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:20:16.201 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:20:18.026 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:22:18.098 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:22:18.140 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:22:19.953 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:22:19.971 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:22:19.971 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:22:19.971 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:22:19.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:22:20.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:22:20.595 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:22:20.662 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:22:20.662 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:22:20.664 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:22:20.853 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:22:20.903 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:22:20.903 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:22:20.903 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:22:20.904 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:22:20.910 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:22:20.954 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:22:22.327 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:24:22.396 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:24:22.441 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:24:24.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:24:24.421 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:24:24.421 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:24:24.421 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:24:24.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:24:24.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:24:25.044 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:24:25.122 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:24:25.123 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:24:25.125 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:24:25.326 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:24:25.386 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:24:25.387 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:24:25.387 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:24:25.387 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:24:25.387 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:24:25.427 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:24:26.531 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:26:26.587 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:26:26.628 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:26:27.705 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:26:27.724 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:26:27.724 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:26:27.724 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:26:27.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:26:28.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:26:28.362 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:26:28.425 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:26:28.425 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:26:28.427 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:26:28.626 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:26:28.674 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:26:28.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:26:28.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:26:28.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:26:28.679 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:26:28.720 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:26:29.787 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:28:29.859 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:28:29.897 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:28:32.123 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:28:32.140 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:28:32.140 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:28:32.140 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:28:32.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:28:32.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:28:32.777 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:28:32.846 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:28:32.846 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:28:32.849 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:28:33.043 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:28:33.091 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:28:33.092 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:28:33.092 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:28:33.092 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:28:33.096 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:28:33.136 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:28:34.505 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:30:34.573 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:30:34.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:30:36.467 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:30:36.486 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:30:36.486 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:30:36.487 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:30:36.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:30:36.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:30:37.121 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:30:37.186 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:30:37.186 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:30:37.189 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:30:37.387 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:30:37.436 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:30:37.436 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:30:37.436 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:30:37.437 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:30:37.444 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:30:37.446 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T10:30:37.446 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T10:30:37.446 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T10:30:37.446 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T10:30:37.447 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T10:30:37.486 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:30:38.560 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:32:38.632 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:32:38.676 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:32:39.710 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:32:39.727 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:32:39.728 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:32:39.728 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:32:39.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:32:40.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:32:40.339 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:32:40.403 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:32:40.403 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:32:40.405 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:32:40.568 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:32:40.609 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T10:32:40.609 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T10:32:40.610 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T10:32:40.610 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T10:32:40.610 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T10:32:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:32:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:32:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:32:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:32:40.627 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:32:40.662 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:32:41.650 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:34:41.715 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:34:41.753 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:34:42.779 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:34:42.796 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:34:42.796 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:34:42.797 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:34:42.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:34:43.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:34:43.394 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:34:43.453 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:34:43.453 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:34:43.455 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:34:43.655 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:34:43.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T10:34:43.694 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T10:34:43.694 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T10:34:43.694 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T10:34:43.694 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T10:34:43.703 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:34:43.703 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:34:43.703 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:34:43.703 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:34:43.708 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:34:43.747 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:34:44.770 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:36:44.837 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:36:44.879 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:36:45.947 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:36:45.975 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:36:45.976 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:36:45.976 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:36:45.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:36:46.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:36:46.604 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:36:46.677 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:36:46.677 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:36:46.679 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:36:46.862 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:36:46.907 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:36:46.907 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:36:46.907 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:36:46.908 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:36:46.912 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:36:46.950 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:36:47.979 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:38:48.047 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:38:48.087 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:38:49.170 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:38:49.188 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:38:49.188 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:38:49.188 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:38:49.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:38:49.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:38:49.813 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:38:49.884 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:38:49.884 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:38:49.886 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:38:50.082 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:38:50.132 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:38:50.132 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:38:50.132 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:38:50.133 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:38:50.137 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T10:38:50.137 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T10:38:50.138 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T10:38:50.138 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T10:38:50.138 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T10:38:50.141 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:38:50.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:38:51.194 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:40:51.258 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:40:51.303 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:40:52.391 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:40:52.408 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:40:52.409 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:40:52.409 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:40:52.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:40:52.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:40:53.019 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:40:53.082 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:40:53.083 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:40:53.085 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:40:53.314 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:40:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:40:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:40:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:40:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:40:53.372 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:40:53.413 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:40:54.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:42:54.573 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:42:54.704 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:42:55.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:42:55.834 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:42:55.834 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:42:55.834 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:42:55.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:42:56.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:42:56.441 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:42:56.513 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:42:56.513 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:42:56.515 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:42:56.723 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:42:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:42:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:42:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:42:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:42:56.780 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:42:56.820 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:42:57.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:44:57.964 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:44:58.005 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:44:59.070 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:44:59.087 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:44:59.088 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:44:59.088 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:44:59.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:44:59.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:44:59.701 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:44:59.772 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:44:59.772 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:44:59.774 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:44:59.966 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:45:00.011 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:45:00.011 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:45:00.011 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:45:00.011 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:45:00.016 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:45:00.055 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:45:01.016 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:47:01.085 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:47:01.126 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:47:02.172 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:47:02.193 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:47:02.193 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:47:02.193 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:47:02.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:47:02.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:47:02.827 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:47:02.893 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:47:02.893 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:47:02.895 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:47:03.088 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:47:03.135 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:47:03.135 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:47:03.135 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:47:03.135 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:47:03.140 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:47:03.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:47:04.161 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:49:04.228 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:49:04.272 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:49:05.333 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:49:05.350 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:49:05.350 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:49:05.350 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:49:05.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:49:05.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:49:05.963 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:49:06.029 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:49:06.030 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:49:06.031 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:49:06.191 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:49:06.233 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T10:49:06.233 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T10:49:06.233 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T10:49:06.233 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T10:49:06.233 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T10:49:06.253 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:49:06.253 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:49:06.253 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:49:06.253 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:49:06.259 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:49:06.299 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:49:07.384 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:51:07.483 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:51:07.540 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:51:08.594 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:51:08.611 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:51:08.611 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:51:08.611 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:51:08.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:51:08.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:51:09.226 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:51:09.297 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:51:09.297 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:51:09.299 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:51:09.502 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:51:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:51:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:51:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:51:09.549 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:51:09.553 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:51:09.595 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:51:10.589 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:53:10.667 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:53:10.712 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:53:11.795 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:53:11.812 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:53:11.812 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:53:11.812 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:53:11.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:53:12.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:53:12.432 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:53:12.507 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:53:12.507 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:53:12.510 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:53:12.716 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:53:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:53:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:53:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:53:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:53:12.771 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:53:12.821 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:53:13.990 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:55:14.058 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:55:14.097 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:55:15.166 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:55:15.186 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:55:15.186 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:55:15.186 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:55:15.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:55:15.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:55:15.808 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:55:15.882 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:55:15.882 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:55:15.884 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:55:16.081 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:55:16.129 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:55:16.129 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:55:16.129 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:55:16.130 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:55:16.139 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:55:16.177 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:55:17.221 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:57:17.291 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:57:17.331 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:57:18.385 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:57:18.403 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:57:18.403 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:57:18.403 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:57:18.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:57:18.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:57:19.006 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:57:19.077 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:57:19.077 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:57:19.080 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:57:19.297 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:57:19.345 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:57:19.346 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:57:19.346 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:57:19.346 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:57:19.355 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:57:19.356 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T10:57:19.356 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T10:57:19.356 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T10:57:19.356 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T10:57:19.356 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T10:57:19.399 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:57:20.414 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:59:20.484 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T10:59:20.525 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:59:21.555 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T10:59:21.571 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T10:59:21.571 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T10:59:21.572 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:59:21.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:59:21.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T10:59:22.183 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T10:59:22.252 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T10:59:22.252 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T10:59:22.254 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:59:22.434 INFO:teuthology.misc:Compressing logs... 2021-03-23T10:59:22.479 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T10:59:22.479 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T10:59:22.479 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T10:59:22.479 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T10:59:22.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T10:59:22.521 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T10:59:23.580 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:01:23.650 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:01:23.696 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:01:24.863 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:01:24.881 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:01:24.881 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:01:24.881 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:01:24.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:01:25.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:01:25.498 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:01:25.567 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:01:25.568 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:01:25.570 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:01:25.757 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:01:25.796 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:01:25.796 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:01:25.796 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:01:25.796 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:01:25.800 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:01:25.841 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:01:26.941 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:03:27.014 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:03:27.059 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:03:28.126 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:03:28.143 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:03:28.143 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:03:28.143 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:03:28.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:03:28.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:03:28.746 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:03:28.815 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:03:28.816 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:03:28.817 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:03:28.997 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:03:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:03:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:03:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:03:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:03:29.048 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:03:29.090 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:03:30.237 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:05:30.306 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:05:30.349 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:05:31.449 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:05:31.466 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:05:31.466 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:05:31.466 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:05:31.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:05:31.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:05:32.071 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:05:32.141 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:05:32.141 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:05:32.143 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:05:32.328 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:05:32.363 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T11:05:32.364 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T11:05:32.364 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T11:05:32.364 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T11:05:32.364 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T11:05:32.377 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:05:32.377 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:05:32.377 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:05:32.377 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:05:32.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:05:32.427 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:05:33.438 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:07:33.504 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:07:33.545 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:07:34.570 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:07:34.588 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:07:34.588 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:07:34.588 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:07:34.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:07:34.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:07:35.203 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:07:35.273 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:07:35.273 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:07:35.275 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:07:35.457 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:07:35.500 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:07:35.500 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:07:35.501 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:07:35.501 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:07:35.506 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:07:35.543 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:07:36.546 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:09:36.616 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:09:36.657 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:09:37.710 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:09:37.728 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:09:37.728 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:09:37.728 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:09:37.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:09:38.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:09:38.368 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:09:38.430 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:09:38.431 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:09:38.433 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:09:38.639 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:09:38.688 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:09:38.688 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:09:38.688 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:09:38.688 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:09:38.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T11:09:38.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T11:09:38.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T11:09:38.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T11:09:38.693 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T11:09:38.694 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:09:38.735 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:09:39.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:11:39.870 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:11:39.909 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:11:40.951 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:11:40.968 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:11:40.968 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:11:40.968 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:11:40.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:11:41.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:11:41.644 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:11:41.713 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:11:41.713 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:11:41.716 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:11:41.922 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:11:41.962 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T11:11:41.963 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T11:11:41.963 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T11:11:41.963 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T11:11:41.963 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T11:11:41.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:11:41.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:11:41.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:11:41.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:11:41.974 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:11:42.019 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:11:43.119 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:13:43.206 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:13:43.248 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:13:44.302 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:13:44.320 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:13:44.320 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:13:44.320 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:13:44.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:13:44.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:13:44.924 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:13:44.989 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:13:44.989 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:13:44.992 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:13:45.205 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:13:45.253 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:13:45.254 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:13:45.254 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:13:45.254 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:13:45.263 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:13:45.302 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:13:46.272 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:15:46.338 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:15:46.376 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:15:47.385 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:15:47.402 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:15:47.402 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:15:47.402 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:15:47.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:15:47.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:15:48.020 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:15:48.090 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:15:48.090 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:15:48.093 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:15:48.269 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:15:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:15:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:15:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:15:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:15:48.325 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:15:48.433 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:15:49.447 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:17:49.513 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:17:49.552 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:17:50.610 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:17:50.628 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:17:50.628 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:17:50.628 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:17:50.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:17:50.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:17:51.296 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:17:51.362 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:17:51.362 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:17:51.364 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:17:51.584 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:17:51.631 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:17:51.632 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:17:51.632 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:17:51.632 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:17:51.638 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:17:51.678 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:17:52.676 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:19:52.743 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:19:52.785 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:19:53.847 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:19:53.863 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:19:53.863 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:19:53.863 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:19:53.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:19:54.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:19:54.500 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:19:54.565 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:19:54.565 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:19:54.567 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:19:54.757 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:19:54.802 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:19:54.802 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:19:54.802 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:19:54.802 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:19:54.807 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:19:54.847 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:19:55.834 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:21:55.908 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:21:55.949 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:21:57.021 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:21:57.036 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:21:57.036 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:21:57.037 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:21:57.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:21:57.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:21:57.661 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:21:57.725 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:21:57.725 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:21:57.727 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:21:57.934 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:21:57.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:21:57.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:21:57.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:21:57.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:21:57.984 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:21:58.021 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:21:59.111 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:23:59.177 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:23:59.223 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:24:00.342 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:24:00.359 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:24:00.359 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:24:00.360 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:24:00.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:24:00.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:24:00.970 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:24:01.037 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:24:01.038 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:24:01.041 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:24:01.229 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:24:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:24:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:24:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:24:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:24:01.279 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:24:01.325 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:24:02.445 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:26:02.513 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:26:02.557 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:26:03.668 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:26:03.686 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:26:03.686 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:26:03.686 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:26:03.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:26:03.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:26:04.327 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:26:04.394 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:26:04.394 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:26:04.397 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:26:04.582 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:26:04.628 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:26:04.629 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:26:04.629 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:26:04.629 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:26:04.633 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:26:04.674 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:26:05.833 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:28:05.902 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:28:05.942 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:28:06.985 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:28:07.002 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:28:07.002 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:28:07.002 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:28:07.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:28:07.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:28:07.645 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:28:07.706 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:28:07.706 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:28:07.708 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:28:07.911 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:28:07.956 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T11:28:07.956 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T11:28:07.957 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T11:28:07.957 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T11:28:07.957 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T11:28:07.959 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:28:07.959 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:28:07.959 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:28:07.959 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:28:07.963 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:28:08.007 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:28:09.029 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:30:09.097 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:30:09.141 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:30:10.205 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:30:10.225 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:30:10.225 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:30:10.225 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:30:10.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:30:10.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:30:10.877 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:30:10.952 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:30:10.952 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:30:10.953 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:30:11.154 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:30:11.206 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:30:11.206 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:30:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:30:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:30:11.211 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:30:11.251 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:30:12.282 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:32:12.348 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:32:12.390 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:32:13.409 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:32:13.428 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:32:13.428 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:32:13.428 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:32:13.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:32:13.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:32:14.041 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:32:14.111 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:32:14.111 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:32:14.114 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:32:14.330 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:32:14.378 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:32:14.378 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:32:14.378 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:32:14.379 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:32:14.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:32:14.421 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:32:15.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:34:15.429 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:34:15.472 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:34:16.557 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:34:16.579 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:34:16.579 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:34:16.579 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:34:16.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:34:16.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:34:17.221 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:34:17.290 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:34:17.291 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:34:17.293 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:34:17.483 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:34:17.533 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:34:17.533 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:34:17.534 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:34:17.534 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:34:17.536 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T11:34:17.536 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T11:34:17.536 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T11:34:17.537 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T11:34:17.537 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T11:34:17.543 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:34:17.584 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:34:18.690 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:36:18.756 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:36:18.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:36:19.905 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:36:19.922 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:36:19.923 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:36:19.923 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:36:19.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:36:20.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:36:20.543 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:36:20.611 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:36:20.611 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:36:20.613 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:36:20.815 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:36:20.862 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:36:20.862 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:36:20.862 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:36:20.862 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:36:20.870 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:36:20.909 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:36:21.958 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:38:22.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:38:22.070 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:38:23.196 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:38:23.213 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:38:23.213 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:38:23.214 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:38:23.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:38:23.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:38:23.824 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:38:23.889 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:38:23.889 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:38:23.892 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:38:24.081 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:38:24.115 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T11:38:24.115 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T11:38:24.115 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T11:38:24.116 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T11:38:24.116 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T11:38:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:38:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:38:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:38:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:38:24.132 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:38:24.175 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:38:25.235 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:40:25.305 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:40:25.346 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:40:26.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:40:26.422 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:40:26.422 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:40:26.422 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:40:26.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:40:26.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:40:27.066 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:40:27.134 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:40:27.134 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:40:27.140 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:40:27.342 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:40:27.400 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T11:40:27.400 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T11:40:27.400 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T11:40:27.400 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T11:40:27.400 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T11:40:27.405 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:40:27.405 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:40:27.405 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:40:27.405 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:40:27.411 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:40:27.455 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:40:28.483 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:42:28.550 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:42:28.590 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:42:29.637 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:42:29.655 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:42:29.656 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:42:29.656 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:42:29.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:42:29.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:42:30.273 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:42:30.341 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:42:30.341 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:42:30.344 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:42:30.544 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:42:30.593 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:42:30.593 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:42:30.593 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:42:30.593 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:42:30.599 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T11:42:30.599 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T11:42:30.599 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T11:42:30.599 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T11:42:30.599 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T11:42:30.602 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:42:30.642 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:42:31.695 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:44:31.764 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:44:31.834 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:44:32.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:44:32.919 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:44:32.919 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:44:32.919 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:44:32.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:44:33.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:44:33.544 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:44:33.609 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:44:33.609 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:44:33.611 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:44:33.811 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:44:33.859 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:44:33.859 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:44:33.860 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:44:33.860 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:44:33.867 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:44:33.912 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:44:35.023 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:46:35.093 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:46:35.134 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:46:36.286 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:46:36.311 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:46:36.311 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:46:36.312 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:46:36.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:46:36.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:46:36.927 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:46:36.987 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:46:36.987 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:46:36.990 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:46:37.193 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:46:37.242 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:46:37.242 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:46:37.243 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:46:37.243 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:46:37.248 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:46:37.291 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:46:38.364 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:48:38.428 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:48:38.476 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:48:39.571 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:48:39.589 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:48:39.589 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:48:39.589 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:48:39.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:48:39.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:48:40.269 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:48:40.335 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:48:40.335 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:48:40.337 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:48:40.517 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:48:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:48:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:48:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:48:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:48:40.575 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:48:40.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:48:41.778 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:50:41.938 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:50:41.980 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:50:43.049 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:50:43.141 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:50:43.141 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:50:43.141 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:50:43.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:50:43.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:50:43.821 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:50:43.889 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:50:43.889 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:50:43.892 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:50:44.072 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:50:44.119 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T11:50:44.120 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T11:50:44.120 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T11:50:44.120 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T11:50:44.120 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T11:50:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:50:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:50:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:50:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:50:44.137 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:50:44.179 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:50:45.253 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:52:45.319 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:52:45.358 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:52:46.372 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:52:46.388 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:52:46.389 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:52:46.389 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:52:46.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:52:46.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:52:47.018 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:52:47.084 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:52:47.084 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:52:47.086 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:52:47.262 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:52:47.308 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:52:47.308 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:52:47.308 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:52:47.308 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:52:47.313 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:52:47.353 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:52:48.412 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:54:48.480 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:54:48.522 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:54:49.593 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:54:49.610 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:54:49.611 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:54:49.611 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:54:49.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:54:49.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:54:50.225 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:54:50.294 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:54:50.294 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:54:50.297 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:54:50.495 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:54:50.534 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:54:50.535 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:54:50.535 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:54:50.535 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:54:50.540 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:54:50.580 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:54:51.599 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:56:51.666 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:56:51.710 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:56:52.776 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:56:52.798 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:56:52.798 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:56:52.799 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:56:52.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:56:53.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:56:53.473 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:56:53.537 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:56:53.538 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:56:53.540 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:56:53.709 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:56:53.754 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:56:53.754 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:56:53.754 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:56:53.755 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:56:53.760 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:56:53.803 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:56:54.893 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:58:54.960 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T11:58:55.004 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:58:56.101 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:58:56.118 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T11:58:56.119 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T11:58:56.119 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:58:56.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:58:56.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T11:58:56.834 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T11:58:56.902 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T11:58:56.903 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T11:58:56.905 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:58:57.107 INFO:teuthology.misc:Compressing logs... 2021-03-23T11:58:57.158 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T11:58:57.159 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T11:58:57.159 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T11:58:57.159 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T11:58:57.176 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T11:58:57.220 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T11:58:58.343 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T11:58:58.347 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T11:58:58.347 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T11:58:58.348 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T11:58:58.348 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T11:58:58.348 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:00:58.419 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:00:58.458 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:00:59.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:00:59.519 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:00:59.519 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:00:59.520 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:00:59.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:00:59.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:01:00.138 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:01:00.208 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:01:00.208 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:01:00.210 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:01:00.414 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:01:00.452 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:01:00.453 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:01:00.453 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:01:00.453 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:01:00.453 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:01:00.463 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:01:00.463 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:01:00.464 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:01:00.464 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:01:00.468 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:01:00.511 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:01:01.628 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:03:01.697 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:03:01.740 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:03:02.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:03:02.892 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:03:02.892 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:03:02.892 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:03:02.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:03:03.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:03:03.524 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:03:03.596 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:03:03.596 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:03:03.599 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:03:03.800 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:03:03.850 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:03:03.850 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:03:03.850 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:03:03.850 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:03:03.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:03:03.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:03:03.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:03:03.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:03:03.853 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:03:03.856 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:03:03.896 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:03:04.978 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:05:05.053 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:05:05.092 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:05:06.230 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:05:06.248 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:05:06.248 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:05:06.248 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:05:06.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:05:06.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:05:06.882 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:05:06.942 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:05:06.942 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:05:06.943 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:05:07.129 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:05:07.174 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:05:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:05:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:05:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:05:07.178 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:05:07.178 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:05:07.178 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:05:07.178 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:05:07.179 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:05:07.180 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:05:07.222 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:05:08.324 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:07:08.450 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:07:08.491 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:07:09.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:07:09.558 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:07:09.558 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:07:09.558 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:07:09.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:07:09.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:07:10.187 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:07:10.257 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:07:10.258 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:07:10.260 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:07:10.459 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:07:10.504 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:07:10.669 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:07:10.670 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:07:10.670 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:07:10.695 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:07:10.738 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:07:11.817 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:09:11.885 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:09:11.927 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:09:12.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:09:13.009 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:09:13.009 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:09:13.009 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:09:13.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:09:13.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:09:13.787 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:09:13.856 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:09:13.856 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:09:13.858 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:09:14.071 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:09:14.113 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:09:14.113 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:09:14.113 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:09:14.114 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:09:14.114 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:09:14.137 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:09:14.137 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:09:14.137 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:09:14.137 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:09:14.143 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:09:14.183 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:09:15.225 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:11:15.290 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:11:15.330 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:11:16.492 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:11:16.509 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:11:16.509 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:11:16.509 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:11:16.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:11:16.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:11:17.133 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:11:17.198 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:11:17.198 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:11:17.200 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:11:17.377 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:11:17.418 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:11:17.418 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:11:17.418 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:11:17.419 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:11:17.422 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:11:17.464 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:11:18.477 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:13:18.540 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:13:18.593 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:13:20.678 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:13:20.696 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:13:20.696 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:13:20.696 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:13:20.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:13:20.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:13:21.339 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:13:21.409 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:13:21.409 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:13:21.412 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:13:21.629 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:13:21.668 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:13:21.668 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:13:21.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:13:21.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:13:21.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:13:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:13:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:13:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:13:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:13:21.693 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:13:21.764 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:13:23.724 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:15:23.794 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:15:23.840 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:15:26.018 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:15:26.041 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:15:26.041 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:15:26.041 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:15:26.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:15:26.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:15:26.641 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:15:26.707 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:15:26.707 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:15:26.710 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:15:26.871 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:15:26.917 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:15:26.917 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:15:26.917 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:15:26.917 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:15:26.917 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:15:26.938 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:15:26.938 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:15:26.938 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:15:26.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:15:26.945 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:15:26.987 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:15:28.847 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:17:28.917 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:17:28.959 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:17:31.115 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:17:31.141 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:17:31.141 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:17:31.142 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:17:31.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:17:31.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:17:31.837 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:17:31.905 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:17:31.905 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:17:31.907 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:17:32.117 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:17:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:17:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:17:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:17:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:17:32.163 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:17:32.210 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:17:33.366 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:19:33.439 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:19:33.482 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:19:36.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:19:36.705 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:19:36.706 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:19:36.706 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:19:36.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:19:37.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:19:37.377 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:19:37.444 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:19:37.444 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:19:37.447 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:19:37.636 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:19:37.683 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:19:37.683 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:19:37.684 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:19:37.684 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:19:37.690 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:19:37.737 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:19:38.880 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:21:38.955 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:21:38.995 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:21:40.814 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:21:40.859 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:21:40.859 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:21:40.860 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:21:40.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:21:41.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:21:41.486 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:21:41.555 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:21:41.555 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:21:41.557 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:21:41.782 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:21:41.834 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:21:41.834 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:21:41.834 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:21:41.835 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:21:41.838 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:21:41.880 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:21:42.937 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:23:43.012 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:23:43.061 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:23:44.200 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:23:44.218 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:23:44.218 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:23:44.218 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:23:44.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:23:44.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:23:44.902 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:23:44.974 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:23:44.974 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:23:44.976 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:23:45.153 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:23:45.208 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:23:45.208 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:23:45.208 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:23:45.209 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:23:45.209 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:23:45.211 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:23:45.211 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:23:45.211 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:23:45.211 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:23:45.218 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:23:45.287 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:23:46.414 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:25:46.486 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:25:46.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:25:47.725 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:25:47.743 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:25:47.743 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:25:47.743 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:25:47.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:25:48.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:25:48.410 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:25:48.478 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:25:48.478 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:25:48.481 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:25:48.645 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:25:48.692 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:25:48.693 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:25:48.693 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:25:48.693 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:25:48.700 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:25:48.742 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:25:49.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:27:49.953 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:27:49.993 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:27:51.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:27:51.043 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:27:51.043 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:27:51.043 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:27:51.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:27:51.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:27:51.710 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:27:51.783 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:27:51.783 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:27:51.785 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:27:51.960 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:27:52.009 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:27:52.009 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:27:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:27:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:27:52.015 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:27:52.056 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:27:53.211 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:29:53.286 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:29:53.327 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:29:54.444 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:29:54.461 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:29:54.461 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:29:54.461 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:29:54.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:29:54.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:29:55.088 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:29:55.159 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:29:55.160 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:29:55.162 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:29:55.352 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:29:55.397 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:29:55.397 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:29:55.397 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:29:55.397 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:29:55.403 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:29:55.444 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:29:56.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:31:56.593 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:31:56.638 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:31:57.682 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:31:57.698 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:31:57.698 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:31:57.699 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:31:57.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:31:57.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:31:58.295 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:31:58.366 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:31:58.366 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:31:58.368 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:31:58.566 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:31:58.610 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:31:58.611 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:31:58.611 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:31:58.611 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:31:58.615 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:31:58.655 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:31:59.701 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:33:59.771 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:33:59.813 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:34:00.951 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:34:00.971 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:34:00.971 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:34:00.971 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:34:00.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:34:01.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:34:01.642 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:34:01.708 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:34:01.709 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:34:01.718 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:34:01.882 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:34:01.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:34:01.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:34:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:34:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:34:01.939 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:34:01.939 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:34:01.939 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:34:01.939 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:34:01.939 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:34:01.940 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:34:01.983 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:34:03.031 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:36:03.099 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:36:03.157 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:36:04.234 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:36:04.266 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:36:04.266 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:36:04.266 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:36:04.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:36:04.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:36:04.898 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:36:04.967 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:36:04.967 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:36:04.969 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:36:05.161 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:36:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:36:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:36:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:36:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:36:05.217 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:36:05.256 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:36:06.277 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:38:06.343 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:38:06.389 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:38:07.452 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:38:07.469 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:38:07.469 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:38:07.470 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:38:07.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:38:07.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:38:08.099 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:38:08.169 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:38:08.169 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:38:08.171 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:38:08.359 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:38:08.408 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:38:08.408 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:38:08.408 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:38:08.408 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:38:08.413 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:38:08.452 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:38:09.557 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:40:09.631 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:40:09.672 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:40:10.734 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:40:10.752 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:40:10.752 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:40:10.752 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:40:10.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:40:11.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:40:11.379 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:40:11.442 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:40:11.442 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:40:11.445 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:40:11.658 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:40:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:40:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:40:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:40:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:40:11.710 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:40:11.751 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:40:12.832 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:42:12.903 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:42:12.944 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:42:14.120 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:42:14.138 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:42:14.138 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:42:14.139 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:42:14.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:42:14.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:42:14.780 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:42:14.844 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:42:14.844 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:42:14.846 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:42:15.042 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:42:15.093 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:42:15.094 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:42:15.094 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:42:15.094 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:42:15.098 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:42:15.098 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:42:15.099 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:42:15.099 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:42:15.099 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:42:15.100 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:42:15.141 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:42:16.183 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:44:16.267 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:44:16.499 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:44:18.432 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:44:18.449 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:44:18.450 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:44:18.450 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:44:18.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:44:18.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:44:19.072 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:44:19.144 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:44:19.144 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:44:19.147 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:44:19.345 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:44:19.391 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:44:19.392 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:44:19.392 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:44:19.392 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:44:19.398 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:44:19.474 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:44:20.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:46:20.586 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:46:20.628 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:46:22.533 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:46:22.550 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:46:22.550 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:46:22.551 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:46:22.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:46:22.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:46:23.162 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:46:23.232 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:46:23.232 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:46:23.234 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:46:23.445 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:46:23.498 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:46:23.498 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:46:23.498 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:46:23.498 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:46:23.507 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:46:23.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:46:23.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:46:23.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:46:23.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:46:23.509 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:46:23.548 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:46:24.564 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:48:24.632 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:48:24.700 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:48:26.535 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:48:26.552 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:48:26.552 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:48:26.552 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:48:26.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:48:26.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:48:27.181 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:48:27.249 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:48:27.250 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:48:27.252 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:48:27.458 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:48:27.505 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:48:27.505 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:48:27.505 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:48:27.505 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:48:27.511 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:48:27.551 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:48:28.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:50:28.622 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:50:28.662 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:50:30.662 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:50:30.682 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:50:30.683 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:50:30.683 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:50:30.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:50:30.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:50:31.313 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:50:31.388 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:50:31.389 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:50:31.391 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:50:31.619 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:50:31.659 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:50:31.659 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:50:31.659 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:50:31.659 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:50:31.659 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:50:31.667 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:50:31.667 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:50:31.667 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:50:31.667 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:50:31.672 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:50:31.713 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:50:32.691 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:52:32.763 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:52:32.833 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:52:34.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:52:34.873 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:52:34.873 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:52:34.873 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:52:34.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:52:35.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:52:35.495 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:52:35.564 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:52:35.565 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:52:35.566 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:52:35.792 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:52:35.846 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T12:52:35.846 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T12:52:35.846 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T12:52:35.846 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T12:52:35.846 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T12:52:35.851 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:52:35.851 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:52:35.851 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:52:35.851 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:52:35.857 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:52:35.899 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:52:36.964 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:54:37.068 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:54:37.109 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:54:39.022 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:54:39.040 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:54:39.041 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:54:39.041 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:54:39.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:54:39.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:54:39.653 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:54:39.720 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:54:39.720 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:54:39.723 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:54:39.919 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:54:39.964 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:54:39.964 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:54:39.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:54:39.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:54:39.976 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:54:40.017 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:54:41.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:56:41.138 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:56:41.210 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:56:43.264 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:56:43.281 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:56:43.281 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:56:43.281 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:56:43.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:56:43.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:56:43.914 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:56:43.981 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:56:43.981 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:56:43.984 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:56:44.182 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:56:44.225 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:56:44.225 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:56:44.225 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:56:44.225 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:56:44.230 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:56:44.271 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:56:45.309 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:58:45.373 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T12:58:45.441 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:58:47.419 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T12:58:47.436 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T12:58:47.436 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T12:58:47.437 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:58:47.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:58:47.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T12:58:48.058 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T12:58:48.128 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T12:58:48.129 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T12:58:48.131 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:58:48.341 INFO:teuthology.misc:Compressing logs... 2021-03-23T12:58:48.391 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T12:58:48.391 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T12:58:48.391 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T12:58:48.391 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T12:58:48.397 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T12:58:48.517 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T12:58:49.727 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:00:49.807 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:00:49.850 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:00:51.915 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:00:51.937 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:00:51.937 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:00:51.937 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:00:51.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:00:52.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:00:52.637 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:00:52.703 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:00:52.703 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:00:52.706 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:00:52.876 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:00:52.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:00:52.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:00:52.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:00:52.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:00:52.937 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T13:00:52.937 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T13:00:52.937 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T13:00:52.937 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T13:00:52.937 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T13:00:52.939 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:00:52.980 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:00:54.668 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:02:54.740 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:02:54.827 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:02:57.109 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:02:57.128 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:02:57.129 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:02:57.129 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:02:57.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:02:57.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:02:57.756 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:02:57.828 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:02:57.829 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:02:57.831 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:02:58.017 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:02:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:02:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:02:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:02:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:02:58.063 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:02:58.109 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:03:00.226 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:05:00.300 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:05:00.351 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:05:01.446 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:05:01.465 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:05:01.465 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:05:01.465 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:05:01.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:05:01.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:05:02.083 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:05:02.157 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:05:02.158 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:05:02.160 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:05:02.344 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:05:02.381 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T13:05:02.382 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T13:05:02.382 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T13:05:02.382 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T13:05:02.382 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T13:05:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:05:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:05:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:05:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:05:02.394 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:05:02.437 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:05:03.490 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:07:03.557 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:07:03.599 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:07:04.705 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:07:04.723 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:07:04.723 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:07:04.724 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:07:04.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:07:04.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:07:05.341 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:07:05.402 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:07:05.402 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:07:05.404 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:07:05.593 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:07:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:07:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:07:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:07:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:07:05.643 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:07:05.684 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:07:06.804 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:09:06.875 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:09:06.919 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:09:08.765 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:09:08.783 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:09:08.783 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:09:08.784 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:09:08.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:09:09.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:09:09.386 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:09:09.457 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:09:09.458 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:09:09.460 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:09:09.688 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:09:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:09:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:09:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:09:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:09:09.747 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:09:09.796 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:09:10.911 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:11:10.987 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:11:11.030 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:11:12.163 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:11:12.186 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:11:12.186 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:11:12.186 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:11:12.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:11:12.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:11:12.821 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:11:12.889 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:11:12.889 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:11:12.891 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:11:13.080 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:11:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:11:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:11:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:11:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:11:13.134 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:11:13.177 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:11:14.284 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:13:14.355 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:13:14.397 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:13:16.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:13:16.518 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:13:16.518 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:13:16.518 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:13:16.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:13:16.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:13:17.165 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:13:17.241 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:13:17.241 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:13:17.243 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:13:17.444 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:13:17.490 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:13:17.490 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:13:17.490 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:13:17.490 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:13:17.495 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:13:17.560 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:13:19.807 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:15:19.875 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:15:19.923 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:15:21.915 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:15:21.935 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:15:21.936 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:15:21.936 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:15:21.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:15:22.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:15:22.551 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:15:22.618 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:15:22.619 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:15:22.620 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:15:22.822 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:15:22.868 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:15:22.868 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:15:22.868 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:15:22.868 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:15:22.875 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:15:22.920 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:15:24.022 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:17:24.116 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:17:24.162 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:17:26.385 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:17:26.402 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:17:26.402 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:17:26.402 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:17:26.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:17:26.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:17:27.024 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:17:27.101 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:17:27.102 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:17:27.104 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:17:27.300 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:17:27.355 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:17:27.355 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:17:27.355 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:17:27.355 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:17:27.355 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:17:27.401 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:17:28.580 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:19:28.651 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:19:28.691 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:19:29.787 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:19:29.805 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:19:29.805 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:19:29.805 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:19:29.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:19:30.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:19:30.437 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:19:30.513 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:19:30.513 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:19:30.516 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:19:30.706 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:19:30.752 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:19:30.752 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:19:30.752 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:19:30.752 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:19:30.757 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:19:30.799 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:19:31.849 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:21:31.918 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:21:31.958 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:21:32.956 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:21:32.973 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:21:32.973 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:21:32.973 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:21:32.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:21:33.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:21:33.591 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:21:33.659 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:21:33.660 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:21:33.662 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:21:33.858 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:21:33.901 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:21:33.901 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:21:33.901 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:21:33.902 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:21:33.908 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:21:33.947 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:21:35.001 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:23:35.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:23:35.135 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:23:37.120 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:23:37.139 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:23:37.139 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:23:37.139 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:23:37.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:23:37.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:23:37.764 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:23:37.837 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:23:37.837 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:23:37.840 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:23:38.039 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:23:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:23:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:23:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:23:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:23:38.092 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:23:38.133 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:23:39.254 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:25:39.323 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:25:39.399 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:25:41.297 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:25:41.314 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:25:41.314 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:25:41.314 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:25:41.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:25:41.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:25:41.944 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:25:42.017 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:25:42.017 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:25:42.019 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:25:42.215 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:25:42.260 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:25:42.260 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:25:42.261 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:25:42.261 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:25:42.268 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:25:42.312 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:25:44.150 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:27:44.241 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:27:44.314 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:27:46.374 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:27:46.393 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:27:46.393 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:27:46.393 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:27:46.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:27:46.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:27:47.028 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:27:47.099 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:27:47.099 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:27:47.102 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:27:47.296 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:27:47.345 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:27:47.345 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:27:47.345 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:27:47.345 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:27:47.350 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:27:47.393 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:27:48.786 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:29:48.862 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:29:49.013 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:29:50.777 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:29:50.797 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:29:50.798 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:29:50.798 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:29:50.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:29:51.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:29:51.419 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:29:51.499 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:29:51.499 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:29:51.502 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:29:51.673 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:29:51.720 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:29:51.720 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:29:51.720 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:29:51.720 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:29:51.727 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:29:51.770 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:29:52.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:31:52.937 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:31:53.010 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:31:54.839 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:31:54.856 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:31:54.857 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:31:54.857 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:31:54.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:31:55.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:31:55.491 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:31:55.560 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:31:55.560 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:31:55.562 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:31:55.741 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:31:55.786 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:31:55.786 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:31:55.786 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:31:55.786 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:31:55.793 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:31:55.835 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:31:57.557 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:33:57.625 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:33:57.779 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:33:59.661 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:33:59.679 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:33:59.679 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:33:59.679 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:33:59.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:33:59.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:34:00.313 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:34:00.381 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:34:00.381 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:34:00.384 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:34:00.578 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:34:00.629 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:34:00.629 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:34:00.629 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:34:00.629 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:34:00.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:34:00.678 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:34:02.376 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:36:02.444 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:36:02.523 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:36:04.399 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:36:04.418 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:36:04.419 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:36:04.419 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:36:04.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:36:04.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:36:05.059 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:36:05.121 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:36:05.121 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:36:05.129 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:36:05.314 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:36:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:36:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:36:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:36:05.359 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:36:05.363 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:36:05.413 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:36:07.321 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:38:07.399 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:38:07.480 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:38:09.582 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:38:09.599 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:38:09.599 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:38:09.599 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:38:09.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:38:09.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:38:10.263 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:38:10.337 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:38:10.337 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:38:10.341 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:38:10.515 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:38:10.568 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:38:10.568 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:38:10.568 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:38:10.568 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:38:10.572 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T13:38:10.573 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T13:38:10.573 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T13:38:10.573 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T13:38:10.573 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T13:38:10.574 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:38:10.619 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:38:12.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:40:12.710 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:40:12.788 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:40:14.677 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:40:14.699 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:40:14.699 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:40:14.700 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:40:14.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:40:14.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:40:15.338 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:40:15.409 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:40:15.409 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:40:15.413 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:40:15.627 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:40:15.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T13:40:15.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T13:40:15.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T13:40:15.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T13:40:15.669 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T13:40:15.694 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:40:15.695 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:40:15.695 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:40:15.695 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:40:15.701 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:40:15.747 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:40:16.924 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:42:16.993 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:42:17.064 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:42:18.116 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:42:18.133 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:42:18.134 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:42:18.134 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:42:18.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:42:18.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:42:18.776 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:42:18.840 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:42:18.840 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:42:18.843 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:42:19.026 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:42:19.066 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:42:19.067 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:42:19.067 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:42:19.067 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:42:19.070 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T13:42:19.070 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T13:42:19.070 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T13:42:19.070 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T13:42:19.070 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T13:42:19.071 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:42:19.117 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:42:20.220 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:44:20.289 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:44:20.368 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:44:22.365 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:44:22.397 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:44:22.398 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:44:22.398 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:44:22.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:44:22.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:44:23.035 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:44:23.107 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:44:23.107 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:44:23.109 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:44:23.308 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:44:23.348 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T13:44:23.349 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T13:44:23.349 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T13:44:23.349 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T13:44:23.349 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T13:44:23.354 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:44:23.354 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:44:23.354 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:44:23.354 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:44:23.361 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:44:23.407 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:44:25.342 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:46:25.416 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:46:25.494 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:46:27.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:46:27.375 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:46:27.375 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:46:27.375 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:46:27.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:46:27.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:46:28.074 INFO:teuthology.orchestra.run.smithi205.stderr:find: '/var/lib/ceph/crash': No such file or directory 2021-03-23T13:46:28.140 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:46:28.141 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:46:28.143 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:46:28.322 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:46:28.361 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2021-03-23T13:46:28.362 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.11.log.gz already exists; not overwritten 2021-03-23T13:46:28.362 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.9.log.gz already exists; not overwritten 2021-03-23T13:46:28.362 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-osd.8.log.gz already exists; not overwritten 2021-03-23T13:46:28.362 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /var/log/ceph/df033248-8b7a-11eb-90a2-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2021-03-23T13:46:28.381 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:46:28.381 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:46:28.381 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:46:28.381 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:46:28.386 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:46:28.430 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:46:30.163 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:48:30.236 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:48:30.330 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:48:32.274 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:48:32.295 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:48:32.296 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:48:32.296 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:48:32.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:48:32.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:48:51.011 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T13:48:51.057 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:48:53.002 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:50:53.071 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:50:53.157 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:50:54.259 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:50:54.277 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:50:54.277 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:50:54.277 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:50:54.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:50:54.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:50:57.666 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T13:50:57.705 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:50:58.766 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:52:58.836 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:52:58.920 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:53:00.830 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:53:00.849 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:53:00.849 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:53:00.849 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:53:00.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:53:01.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:53:01.243 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-23T13:53:02.340 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-23T13:53:03.441 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-23T13:53:04.667 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-23T13:53:05.772 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-23T13:53:06.875 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-23T13:53:07.997 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-23T13:53:09.100 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-23T13:53:10.197 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-23T13:53:11.254 ERROR:teuthology.orchestra.connection:Error authenticating with smithi205.front.sepia.ceph.com: Authentication failed. 2021-03-23T13:53:11.255 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T13:53:11.301 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:53:13.101 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:55:13.170 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:55:13.237 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:55:15.107 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:55:15.126 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:55:15.126 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:55:15.126 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:55:15.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:55:15.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:55:15.799 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T13:55:15.890 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:55:15.890 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:55:15.891 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:55:15.957 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T13:55:16.070 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T13:55:16.070 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:55:16.071 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:55:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:55:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:55:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:55:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:55:16.122 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:55:16.164 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:55:17.641 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:55:17.715 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/ Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 498, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 463, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 320, in report_job response.raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/sage-2021-03-18_16:32:54-rados:cephadm:dashboard-cephadm-iscsi-test-distro-basic-smithi/jobs/5977701/ 2021-03-23T13:57:17.718 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:57:18.196 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:57:20.050 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:57:20.068 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:57:20.068 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:57:20.068 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:57:20.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:57:20.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:57:20.579 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T13:57:20.657 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:57:20.657 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:57:20.658 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:57:20.725 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T13:57:20.843 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T13:57:20.843 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:57:20.844 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:57:20.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:57:20.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:57:20.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:57:20.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:57:20.891 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:57:20.933 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:57:21.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:59:22.063 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T13:59:22.412 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:59:24.422 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T13:59:24.440 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T13:59:24.440 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T13:59:24.440 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:59:24.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:59:24.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T13:59:25.643 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T13:59:25.868 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T13:59:25.868 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:59:25.869 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:59:25.903 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T13:59:26.058 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T13:59:26.059 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T13:59:26.061 INFO:teuthology.misc:Compressing logs... 2021-03-23T13:59:26.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T13:59:26.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T13:59:26.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T13:59:26.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T13:59:26.112 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T13:59:26.152 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T13:59:27.454 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:01:27.538 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:01:27.609 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:01:29.738 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:01:29.756 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:01:29.756 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:01:29.756 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:01:29.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:01:30.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:01:30.335 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T14:01:30.413 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T14:01:30.413 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T14:01:30.415 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:01:30.605 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:01:30.655 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/log/ceph’: No such file or directory 2021-03-23T14:01:30.760 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T14:01:30.760 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T14:01:30.761 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T14:01:30.761 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T14:01:30.761 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:01:30.808 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:01:33.037 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:03:33.135 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:03:33.223 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:03:35.792 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:03:35.811 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:03:35.811 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:03:35.812 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:03:35.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:03:36.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:03:36.419 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T14:03:36.501 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T14:03:36.643 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T14:03:36.666 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:03:36.732 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:03:36.733 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:03:36.778 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:03:39.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:05:39.680 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:05:39.755 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:05:41.744 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:05:41.762 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:05:41.762 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:05:41.762 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:05:41.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:05:42.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:05:42.417 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T14:05:42.537 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T14:05:42.538 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T14:05:42.539 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:05:42.604 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:05:42.623 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:05:42.672 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:05:44.882 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:07:44.956 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:07:45.028 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:07:47.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:07:47.088 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:07:47.089 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:07:47.089 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:07:47.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:07:47.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:07:47.696 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:07:47.764 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:07:47.766 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:07:47.898 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:07:49.839 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:09:49.910 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:09:49.996 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:09:51.848 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:09:51.866 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:09:51.866 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:09:51.866 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:09:51.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:09:52.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:09:52.477 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:09:52.545 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:09:52.547 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:09:52.588 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:09:54.240 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:11:54.311 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:11:54.384 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:11:56.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:11:56.485 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:11:56.486 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:11:56.486 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:11:56.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:11:56.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:11:57.122 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:11:57.189 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:11:57.191 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:11:57.236 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:11:59.258 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:13:59.335 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:13:59.408 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:14:01.270 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:14:01.287 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:14:01.288 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:14:01.288 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:14:01.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:14:01.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:14:02.041 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:14:02.110 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:14:02.111 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:14:02.155 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:14:03.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:16:03.893 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:16:03.964 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:16:05.888 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:16:05.908 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:16:05.908 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:16:05.908 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:16:05.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:16:06.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:16:06.575 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:16:06.643 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:16:06.645 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:16:06.697 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:16:08.629 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:18:08.699 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:18:08.777 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:18:10.793 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:18:10.810 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:18:10.810 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:18:10.810 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:18:10.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:18:11.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:18:11.490 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:18:11.568 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:18:11.569 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:18:11.620 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:18:13.788 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:20:13.858 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:20:13.945 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:20:15.895 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:20:15.913 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:20:15.913 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:20:15.913 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:20:15.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:20:16.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:20:16.532 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:20:16.656 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:20:16.668 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:20:16.713 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:20:17.866 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:22:17.941 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:22:18.038 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:22:20.060 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:22:20.079 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:22:20.079 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:22:20.080 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:22:20.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:22:20.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:22:20.693 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:22:20.760 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:22:20.762 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:22:20.807 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:22:21.963 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:24:22.093 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:24:22.168 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:24:24.092 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:24:24.109 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:24:24.109 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:24:24.109 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:24:24.109 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:24:24.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:24:24.725 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:24:24.792 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:24:24.794 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:24:24.836 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:24:26.536 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:26:26.604 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:26:26.677 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:26:28.585 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:26:28.603 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:26:28.603 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:26:28.603 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:26:28.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:26:28.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:26:29.220 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:26:29.285 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:26:29.287 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:26:29.336 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:26:31.211 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:28:31.282 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:28:31.360 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:28:33.124 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:28:33.142 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:28:33.142 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:28:33.142 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:28:33.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:28:33.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:28:33.758 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:28:33.823 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:28:33.825 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:28:33.874 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:28:35.825 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:30:35.890 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:30:35.958 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:30:38.193 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:30:38.229 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:30:38.229 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:30:38.230 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:30:38.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:30:38.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:30:38.841 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:30:38.907 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:30:38.918 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:30:38.968 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:30:40.779 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:32:40.853 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:32:40.894 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:32:42.968 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:32:42.991 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:32:42.992 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:32:42.992 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:32:42.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:32:43.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:32:43.607 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:32:43.674 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:32:43.676 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:32:43.723 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:32:45.373 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:34:45.449 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:34:45.515 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:34:47.452 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:34:47.470 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:34:47.471 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:34:47.471 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:34:47.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:34:47.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:34:48.059 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:34:48.127 INFO:teuthology.orchestra.run.smithi205.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2021-03-23T14:34:48.128 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi205 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:34:48.169 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:34:49.251 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:36:49.330 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:36:49.374 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:36:51.090 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:36:51.107 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:36:51.107 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:36:51.107 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:36:51.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:36:51.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:37:09.703 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T14:37:09.744 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:37:10.822 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:39:10.888 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:39:10.929 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:39:12.060 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:39:12.077 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:39:12.077 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:39:12.077 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:39:12.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:39:12.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:39:14.055 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T14:39:14.093 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:39:15.167 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:41:15.243 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:41:15.302 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:41:16.365 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:41:16.383 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:41:16.383 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:41:16.383 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:41:16.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:41:16.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:41:19.746 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T14:41:19.786 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:41:20.876 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:43:20.947 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:43:21.023 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:43:22.963 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:43:22.980 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:43:22.981 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:43:22.981 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:43:22.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:43:23.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:43:23.555 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T14:43:23.673 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T14:43:23.673 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T14:43:23.675 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:43:23.705 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T14:43:23.803 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T14:43:23.804 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T14:43:23.804 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:43:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T14:43:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T14:43:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T14:43:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T14:43:23.866 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:43:23.989 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:43:25.850 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:45:25.930 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:45:26.012 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:45:28.291 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:45:28.316 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:45:28.317 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:45:28.317 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:45:28.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:45:28.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:45:28.891 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T14:45:28.978 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T14:45:28.978 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T14:45:28.980 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:45:29.046 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T14:45:29.180 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T14:45:29.180 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T14:45:29.183 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:45:29.234 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T14:45:29.234 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T14:45:29.235 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T14:45:29.235 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T14:45:29.240 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:45:29.309 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:45:31.784 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:47:31.875 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:47:32.048 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:47:34.088 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:47:34.106 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:47:34.106 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:47:34.106 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:47:34.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:47:34.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:47:34.621 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T14:47:34.703 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T14:47:34.703 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T14:47:34.704 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:47:34.769 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2021-03-23T14:47:34.879 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2021-03-23T14:47:34.879 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T14:47:34.880 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:47:34.926 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T14:47:34.927 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T14:47:34.927 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T14:47:34.927 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T14:47:34.931 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:47:34.973 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:47:37.289 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:49:37.361 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:49:37.433 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:49:39.410 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:49:39.432 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:49:39.432 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:49:39.433 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:49:39.433 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:49:39.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:49:42.819 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-23T14:49:42.872 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:49:44.802 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:51:44.877 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-23T14:51:44.946 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:51:46.972 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:51:46.990 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2021-03-23T14:51:46.991 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['host.b', 'client.1'] 2021-03-23T14:51:46.991 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:51:46.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:51:47.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-23T14:51:47.529 INFO:teuthology.orchestra.run.smithi205.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-03-23T14:51:47.648 INFO:teuthology.orchestra.run.smithi205.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-03-23T14:51:47.648 INFO:teuthology.orchestra.run.smithi205.stderr:tar: Error is not recoverable: exiting now 2021-03-23T14:51:47.650 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:51:47.799 INFO:teuthology.misc:Compressing logs... 2021-03-23T14:51:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2021-03-23T14:51:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2021-03-23T14:51:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2021-03-23T14:51:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2021-03-23T14:51:47.847 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2021-03-23T14:51:47.890 INFO:teuthology.kill:Killing Pids: {11609} 2021-03-23T14:51:49.433 INFO:teuthology.kill:No locked machines. Not nuking anything 2021-03-23T14:53:49.559 ERROR:teuthology.dispatcher.supervisor:Child exited with code -9 2021-03-23T14:53:49.679 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-23T14:53:49.680 INFO:teuthology.nuke:Checking targets against current locks 2021-03-23T14:53:49.764 INFO:teuthology.nuke:Not nuking smithi205.front.sepia.ceph.com because description doesn't match 2021-03-23T14:53:49.764 INFO:teuthology.nuke:Not nuking smithi060.front.sepia.ceph.com because description doesn't match